#Snippets

Topics

April 02, 2024

#Enable Intel integrated GPUs on Linux systems

Several misconfigurations may stop a Linux machine from having access to its integrated Intel GPU devices. Let’s check the most common problems and how to fix them.

Firstly, verify that the functionality is enabled on your BIOS. You will likely find relevant settings in the “Chipset” related sections. But if that doesn’t help, it’s time to play around with the kernel.

Make sure that the kernel module i915 isn’t blacklisted. It’s responsible for activating Intel drivers.

# If this returns anything like "blacklist i915", remove the line and reboot
cat /etc/modprobe.d/* | grep i915

Next thing, check your boot Grub configuration. Look for GRUB_CMDLINE_LINUX_DEFAULT and ensure it doesn’t mention nomodeset anywhere. Be aware, however, that removing it might disable HDMI output on your system, so prepare to use SSH access just in case.

cat /etc/default/grub* | grep GRUB_CMDLINE_LINUX_DEFAULT

# If you needed to change something, update the final configuration file
grub-mkconfig -o /boot/grub/grub.cfg

Finally, it’s possible that your iGPU device is not supported by the kernel version you’re currently running with. For instance, as of the time of writing, the default kernel for Ubuntu 22.04.4 LTS Server is 5.15, which doesn’t support 12th-generation Intel iGPUs (e.g., Alder Lake N95 and N100). In such a case, you need a newer kernel.

Ubuntu users can leverage the LTS Enablement Stacks to solve the problem.

apt install --install-recommends linux-generic-hwe-20.04

Intel iGPUs show up on Linux systems under /dev/dri.

# List the available iGPU-related devices
ls -la /dev/dri

# Check iGPU usage
apt install intel-gpu-tools
intel_gpu_top

October 12, 2022

#Troubleshoot CORS related issues on API requests

development server-side kubernetes

Cross Origin Resource Sharing (a.k.a. CORS) is a powerful, and yet misunderstood, web standard for protecting web APIs from abuse. If you’re anything like me, however, you had your fair share of wasted work hours trying to deal with it from time to time. The concept seems deceptively simple, but the devil is in the details.

First of all, its important to understand that CORS compliance is an optional standard and is mostly enforced by browsers, like Chrome, Firefox and Safari. Therefore, it’s unlikely that you will face issues with it while working with libraries like Python’s Requests or CLI tools like curl.

However, if you have a frontend application running on your browser with a domain like, let’s say, http://localhost:3000, API calls directed to different domains, like https://laury.dev, will be preceded by a preflight request. This determines if the browser will execute the main request or not.

When having problems with this process, you may see an error similar to this on your Chrome console:

Access to XMLHttpRequest at ‘https://laury.dev from origin ‘http://localhost:3000 has been blocked by CORS policy: No ‘Access-Control-Allow-Origin’ header is present on the requested resource.

To reproduce the problem in a more isolated manner, you can use curl to make a preflight request manually. The important parts here are:

the HTTP method has to be OPTIONS
three headers must be present: Access-Control-Request-Method, Access-Control-Request-Headers and Origin. Be wary, however, that the domain stated on Origin must mention protocol, FQDN, port (if any is being used), and no trailing slash.

Here’s an example:

curl -IX OPTIONS https://google.com \
    -H "Access-Control-Request-Method: POST" \
    -H "Access-Control-Request-Headers: origin, x-requested-with" \
    -H "Origin: http://mywebsite.com:3000"

For a preflight request to be considered successful, all of the following must be true:

the HTTP status on the response is within 200-299
the response contains these headers: Access-Control-Allow-Methods, Access-Control-Max-Age and Access-Control-Allow-Origin. One extra catch here is that this last header cannot present the value * when the main request contains authentication information (like cookies). If it does, the preflight will be considered a failure by the browser. In this case, the header has to mention the origin domain explicitly.

Kubernetes users running Ingress Nginx

To manage CORS headers on Ingress Nginx, you may use these annotations on Ingress objects. Even objects that only manage paths inside domains will work with them, so there’s no need to enable CORS on the entire domain if you don’t feel like it.

#Use iptables to allow only Cloudflare HTTP requests to target your server

server-side homelab

If you’re using Cloudflare to protect HTTP endpoints, it might be worth it to block any web traffic that doesn’t come from their servers.

To do that, let’s use iptables. The following commands will do the trick, but take note that these IPs might change in the future.

# Allow IPv4 traffic
iptables -I INPUT -p tcp -m multiport --dports http,https -s 103.21.244.0/22 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 103.22.200.0/22 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 103.31.4.0/22 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 104.16.0.0/13 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 104.24.0.0/14 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 108.162.192.0/18 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 131.0.72.0/22 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 141.101.64.0/18 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 162.158.0.0/15 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 172.64.0.0/13 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 173.245.48.0/20 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 188.114.96.0/20 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 190.93.240.0/20 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 197.234.240.0/22 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 198.41.128.0/17 -j ACCEPT

# Allow IPv6 traffic
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2400:cb00::/32 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2606:4700::/32 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2803:f800::/32 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2405:b500::/32 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2405:8100::/32 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2a06:98c0::/29 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2c0f:f248::/32 -j ACCEPT

Now, block everything else on ports 80 and 443.

iptables -A INPUT -p tcp -m multiport --dports http,https -j DROP
ip6tables -A INPUT -p tcp -m multiport --dports http,https -j DROP

Test your new rules and save your changes to persist them across reboots.

netfilter-persistent save

#Basic survival guide and commands for iptables

server-side survival-guide

iptables is a Linux firewall tool that manages packet routing and can block or allow traffic based on rules like a packet’s origin or destination.

Its settings are organized in tables which contain sets of rules, called chains, that will filter data packets.

# Lists existing rules with the "specification" syntax
iptables -S

# Delete a rule by its specification string
iptables -D <spec>

# List existing rules by chain and alongside packet matching statistics
iptables -L -v --line-numbers

# Delete a rule by its row index
iptables -D <chain> <num>

# Reset the packet counters on all chains
iptables -Z

# Accept all requests from an IP
iptables -I INPUT -s <ip> -j ACCEPT

# Drop all requests from an IP
iptables -I INPUT -s <ip> -j DROP

# Accept all incoming TCP traffic to a given port
iptables -I INPUT -m state --state NEW -p tcp --dport <port> -j ACCEPT

# Drop all requests from a range of IPs
iptables -I INPUT -m iprange --src-range <ip_range_start>-<ip_range_end> -j DROP

# Drop all traffic (must be run after acceptance rules; `-A` stands for "append")
iptables -A INPUT -j DROP

# Delete all current rules
iptables -F

# Restore iptables rules from last persisted version
iptables-restore < /etc/iptables/rules.v4

# Persist changes to disk
/sbin/iptables-save
# Depending on your setup, you may need to run another command instead
netfilter-persistent save

References

How the Iptables Firewall Works

How To List and Delete Iptables Firewall Rules

June 26, 2022

#Protect public URLs with ModSecurity for Nginx on a Kubernetes cluster

kubernetes server-side homelab

If you’re using Ingress Nginx to manage public endpoints, ModSecurity is already installed but disabled by default. To enable it, add the following to the ConfigMap ingress-nginx-controller:

allow-snippet-annotations: "true"
enable-modsecurity: "true"
enable-owasp-modsecurity-crs: "true"

To customize the Paranoia and Anomaly Score threshold, you must change rules 900000 and 900110. But remember that higher Paranoia levels lead to stricter rules, and a lower anomaly threshold leads to a more tolerant setup.

modsecurity-snippet: |-
    SecRuleEngine On
    SecAuditEngine Off #  Use RelevantOnly for debugging
    SecAuditLog /dev/stdout
    SecAuditLogFormat JSON
    SecAuditLogType Serial
    SecAuditLogParts ABIJDEFHZ
    SecDebugLog /dev/stdout
    SecDebugLogLevel 3
    SecAction "id:900000,phase:1,nolog,pass,t:none,setvar:tx.paranoia_level=5"
    SecAction "id:900110,phase:1,nolog,pass,t:none,setvar:tx.inbound_anomaly_score_threshold=4,setvar:tx.outbound_anomaly_score_threshold=3"
    Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

Customize ModSecurity rules on a per-host basis

It’s possible to adjust the ModSecurity configuration for each Ingress object on the cluster using annotations.

annotations:
  nginx.ingress.kubernetes.io/enable-modsecurity: "true"
  nginx.ingress.kubernetes.io/enable-owasp-core-rules: "false"
  nginx.ingress.kubernetes.io/modsecurity-snippet: |
    SecRuleRemoveById <rule_id>

How to completely customize modsecurity.conf

With the modsecurity-snippet option, adding custom configuration to ModSecurity is possible. However, you can override the modsecurity.conf entirely if you want.

To do so, first copy the file inside the pod.

kubectl -n ingress-nginx cp <ingress-controller-pod-name>:/etc/nginx/modsecurity/modsecurity.conf ./modsecurity.conf

Modify it as required and save the file in a ConfigMap.

kubectl -n ingress-nginx create configmap modsecurityconf --from-file=modsecurity.conf

If you need to change the file again locally, you can update the ConfigMap by doing the following.

kubectl -n ingress-nginx create configmap modsecurityconf \
  --from-file=modsecurity.conf -o yaml \
  --dry-run=client | kubectl apply -f -

To mount the ConfigMap to the controller deployment, create a patch file.

spec:
  template:
    spec:
      volumes:
      - name: modsecurityconf
        configMap:
          name: modsecurityconf
      containers:
      - name: controller
        volumeMounts:
        - name: modsecurityconf
          mountPath: "/etc/nginx/modsecurity/modsecurity.conf"
          subPath: modsecurity.conf
          readOnly: true

And then apply the patch file.

kubectl -n ingress-nginx patch deployment/ingress-nginx-controller --patch-file deployment-patch.yml

#Host publicly accessible web URLs from a home server using a remote proxy

kubernetes homelab raspberry-pi server-side

Ideally, you would be able to open ports 80 and 443 of your public IP address and update DNS entries with solutions like ddclient or DNS-O-Matic.

However, if your ISP blocks direct traffic to those ports, using a remote server and SSH tunneling can bypass such limitations. With this setup, you can use Kubernetes ingresses as usual. From the ingress perspective, HTTP traffic will reach your home server directly. Therefore, tooling like Nginx and Cert Manager will work with no unique tweaks.

Any Linux box with a public IP can be your remote proxy, but Oracle Cloud has a generous free tier and can be a good starting option. Remember that the smallest available machine is probably enough for your needs since it will only forward traffic and not act on it.

After setting up the remote server with your provider, ensure it has an SSH server running. We’re going to use it to forward network packets for us.

On your remote instance, edit the /etc/ssh/sshd_config file and add GatewayPorts yes to it. Apply the new configuration with service sshd restart.

If you opted for an Oracle Cloud server, go to the Ingress Rules section of the Virtual Cloud Network panel and open ports 80 and 443. Also, adjust iptables in the instance to allow public traffic to ports 80 and 443.

iptables -I INPUT 6 -m state --state NEW -p tcp --dport 80 -j ACCEPT
iptables -I INPUT 6 -m state --state NEW -p tcp --dport 443 -j ACCEPT
netfilter-persistent save

From your home server, you can run a quick test, as long as you already have a HTTP server running locally. Don’t forget to generate a new SSH key for your local server and authorize it on the remote machine. If everything went smoothly until now, you can access a local web endpoint using the remote server IP as the DNS entry for your domain.

ssh -N -R 80:<local_ip>:80 root@<remote_ip>

If the port forwarding worked, you can now create deployment files so that Kubernetes can keep the tunnel running. Customize and apply the following to your local cluster.

apiVersion: v1
kind: Namespace
metadata:
  name: proxy-router

--
apiVersion: apps/v1
kind: Deployment
metadata:
  name: autossh-80
  namespace: proxy-router
  labels:
    app: autossh-80
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  selector:
    matchLabels:
      app: autossh-80
  template:
    metadata:
      labels:
        app: autossh-80
    spec:
      containers:
      - name: autossh-80
        image: jnovack/autossh:2.0.1
        env:
          - name: SSH_REMOTE_USER
            value: "root"
          - name: SSH_REMOTE_HOST
            value: "<remote_server_ip>"
          - name: SSH_REMOTE_PORT
            value: "22"
          - name: SSH_TUNNEL_PORT
            value: "80"
          - name: SSH_BIND_IP
            value: "0.0.0.0"
          - name: SSH_TARGET_HOST
            value: "<local_server_ip>"
          - name: SSH_TARGET_PORT
            value: "80"
          - name: SSH_MODE
            value: "-R"
        volumeMounts:
          - name: keys
            mountPath: /id_rsa
      nodeName: <node_with_ingress_enabled>
      hostNetwork: true
      volumes:
        - name: keys
          hostPath:
            path: <node_path_for_ssh_keys>
            type: File

To redirect port 443, create another deployment using the previous one as a reference.

Any public URLs should have their DNS entries pointing to the remote server’s IP.

August 20, 2020

#Issue Let's Encrypt certificates for domains in a Kubernetes cluster

server-side kubernetes homelab

We will use cert-manager to issue HTTPS certificates for domains served publicly by a Kubernetes cluster.

The application can be installed via Helm, but it’s recommended to install CRDs (Custom Resource Definitions) separately. That may come in handy if you need to delete cert-manager without losing already issued certificates. Don’t forget to change the “version” accordingly on all commands.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.1/cert-manager.crds.yaml

Add JetStack charts to your Helm client.

helm repo add jetstack https://charts.jetstack.io --force-update
helm repo update

Install the application.

helm upgrade --install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.14.1 \
  --set ingressShim.defaultIssuerName=letsencrypt-prod \
  --set ingressShim.defaultIssuerKind=ClusterIssuer \
  --set ingressShim.defaultIssuerGroup=cert-manager.io

To issue actual certificates, you will need a production Issuer or ClusterIssuer. The following yaml defines a ClusterIssuer and assumes that you have a nginx ingress controller. If it suit your needs, apply it to your cluster.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: <your_email>
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx

Verify if your set up is working.

$ kubectl get clusterissuers -A
NAME               READY   AGE
letsencrypt-prod   True    30m

The cert-manager stack will act upon ingresses that comply with a couple of conditions to generate HTTPS certificates. The following example illustrates that. Take special note of the annotations and the “secretName” property under “tls”.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/tls-acme: "true"
  name: nginx-ingress
  namespace: ingress-testing
spec:
  rules:
  - host: <public_URL>
    http:
      paths:
      - backend:
          serviceName: <service_name>
          servicePort: 80
  tls:
  - hosts:
    - <public_URL>
    secretName: <secret_name_of_your_choosing>

#Pretty-print JSON files using the command line

server-side

If you have either Python 2 or 3 installed, use the “json.tool” module to pretty-print.

python -m json.tool <json_file> | less

You can also stream input to Python instead of passing a file directly.

cat <json_file> | python -m json.tool | less

This may be useful for viewing large minified JSON files that could slow down GUI applications.

August 03, 2020

#Quick usage examples of "find" on the command line

server-side

Search recursively for file names using a regex pattern. Use -name for a case sensitive search.

find . -iname '*<partial_name>*'

To exclude results located in certain paths, use -not -path:

find . -iname '*<partial_name>*' -not -path "*/.venv/*"

Execute a command with every result found.

find . -iname '*.md' -type f -exec ls -lh {} \;

List all files in the current folder (recursively) that contain every string mentioned in the search parameters. You can chain as many search terms as you want. Just remember that the last grep needs to have -l as a parameter instead of -q.

find . -type f -exec grep -q '<string1>' {} \; -exec grep -q '<string2>' {} \; -exec grep -l '<string3>' {} \;

Note: every command needs to be terminated by ; or +. But these signs may need to be escaped as in ; or \;. Outputs given by find will be placed on {}. Multiple uses of -exec are allowed.

August 02, 2020

#Read Kubernetes API data using Golang

kubernetes golang development server-side

Get the API client.

import (
  "fmt"
  "context"
  "k8s.io/client-go/kubernetes"
  "k8s.io/client-go/tools/clientcmd"
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// use the current context in kubeconfig
config, err := clientcmd.BuildConfigFromFlags("", "<path_to_KUBECONFIG>")
if err != nil {
  panic(err.Error())
}

// create the clientset
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
  panic(err.Error())
}

Usage examples:

read data about servers

nodeList, err := clientset.CoreV1().Nodes().List(context.TODO(), metav1.ListOptions{})
if err != nil {
  panic(err.Error())
}

read data about pods

namespace := "example"  // leave empty to get data from all namespaces
podList, err := clientset.CoreV1().Pods(namespace).List(context.TODO(), metav1.ListOptions{})
if err != nil {
  panic(err.Error())
}

In the above examples, ListOptions can take two important strings, field selectors and label selectors. This can be used for filtering results.

metav1.ListOptions{
  LabelSelector: "labelName=labelKey",
  FieldSelector: "spec.nodeName=<node_name>",  // Example for filtering by node name
}

Field selectors and label selectors on this case operate in the same way that the CLI options for kubectl do. So, the same rules apply here.

#Read Kubernetes metrics-server data using Golang

kubernetes golang development server-side

First, get the metrics API client.

import (
  "fmt"
  "context"
  "k8s.io/client-go/kubernetes"
  "k8s.io/client-go/tools/clientcmd"
  metricsv "k8s.io/metrics/pkg/client/clientset/versioned"
)

// Get the config
config, err := clientcmd.BuildConfigFromFlags("", "<path_to_KUBECONFIG>")
if err != nil {
  panic(err.Error())
}

// Get the metrics client
metricsClientset, err := metricsv.NewForConfig(config)
if err != nil {
  panic(err.Error())
}

Get the data and store it in podMetricsList array.

namespace := "example"  // leave empty to get data from all namespaces
podMetricsList, err := metricsClientset.MetricsV1beta1().PodMetricses(namespace).List(context.TODO(), metav1.ListOptions{})
if err != nil {
  panic(err.Error())
}

Iterate over the results found.

for _, v := range podMetricsList.Items {
  fmt.Printf("%s\n", v.GetName())
  fmt.Printf("%s\n", v.GetNamespace())
  fmt.Printf("%vm\n", v.Containers[0].Usage.Cpu().MilliValue())
  fmt.Printf("%vMi\n", v.Containers[0].Usage.Memory().Value()/(1024*1024))
}

Note: if you’re inspecting pods that may have more than one container, you’ll need to iterate over v.Containers as well.

August 01, 2020

#Receive data from Kubernetes API using curl

server-side kubernetes

Start by running kubectl’s proxy. With its help, we avoid dealing with authentication headers.

kubectl proxy

Test the connection with a basic call.

curl http://localhost:8001/api/

To check the API groups available, access the root.

curl http://localhost:8001/apis/

Based on the group listing, you can infer URLs for resources.

For cluster-scoped resources, use:

/apis/<GROUP>/<VERSION>/<RESOURCETYPE>
/apis/<GROUP>/<VERSION>/<RESOURCETYPE>/<NAME>

And, for namespace-scoped resources:

/apis/<GROUP>/<VERSION>/<RESOURCETYPE>
/apis/<GROUP>/<VERSION>/namespaces/<NAMESPACE>/<RESOURCETYPE>
/apis/<GROUP>/<VERSION>/namespaces/<NAMESPACE>/<RESOURCETYPE>/<NAME>

To get deployment data, for example, the API Group is “apps”, Version is “v1” and Resourcetype is “deployments”. The final request looks like this:

curl http://localhost:8001/apis/apps/v1/deployments

If your Kubernetes cluster has a metrics-server, the following request will get node resource usage data:

curl http://localhost:8001/apis/metrics.k8s.io/v1beta1/nodes

Rancher users

However, if your cluster was created using Rancher, you’ll need “Authorized Cluster Endpoint” enabled, so your requests can reach the API Server directly. Your URL paths will slightly differ from the examples above otherwise.

Using your command line, you’ll also need to select the correct context for kubectl before activating the proxy.

# List available contexts
kubectl config get-contexts

# Select the direct API context
# Its name has the format <cluster_name>-<pilot_name>
kubectl config use-context <direct_context>

#Activate the replicaset mode in MongoDB

server-side mongodb

Start the MongoDB daemon with the --replSet flag.

mongod --replSet rs0

After first start, initialize the replica set with the following:

mongo <mongo_host>/<database> --eval "rs.initiate({_id: 'rs0', members: [ { _id: 0, host: '<mongo_host>:27017' } ]})"

July 23, 2020

#List Kernel modules installed in a Linux machine

server-side

Get a nicely formatted list:

lsmod

Get information about a specific module (if the module is not installed, this command will fail):

modinfo <mod_name>

July 23, 2020

#Test networking performance between two Linux machines

server-side

Let’s use iperf to measure networking performance. First, set a “server” machine.

iperf -s

From a “client” machine, run the test.

iperf -c <server_ip>

If you are having trouble getting the server’s IP, use ip addr show to find addresses used by the current machine in any connected networks.

July 23, 2020

#Wipe any filesystem data and partitions from devices on Linux

server-side

wipefs should work in most cases to clean filesystems.

# View filesystem data from device
wipefs /dev/<my_device>

# Wipe any filesystem found
wipefs --all /dev/<my_device>

If wipefs can’t detect a filesystem, you can quickly wipe the header data of the device with dd.

dd if=/dev/zero of=/dev/<my_device> bs=1M count=100 oflag=direct,dsync status=progress

You may need to restart your machine to make the filesystem deletion known to the OS, if your device is a “LVM2_member” for instance.

Also, as a last resort, you can completely wipe the device.

dd if=/dev/zero of=/dev/<my_device> bs=1M count=5000000 oflag=direct,dsync status=progress

To delete partitions, sgdisk should work well.

sgdisk --zap-all /dev/<my_device>

July 16, 2020

#Download files recursively through a FTP connection using wget

server-side

Unfortunately, SCP and Rsync aren’t always an option to get our files out of a server. When FTP is all we have, wget can still do a pretty good job and not re-download files you already have, like a primitive Rsync.

wget -nc -nH --cut-dirs=<NUMBER_REMOTE_FOLDERS_TO_CUT_FROM_PATH> --ftp-user='<FTP_USER>' --ftp-password='<FTP_PASSWORD>' -rl 0 -P <LOCAL_PATH> 'ftp://<REMOTE_HOST>/<PATH_TO_FILES>'

Explanation for the used flags:

-nc, --no-clobber                skip downloads that would download to existing files (overwriting them)
-nH, --no-host-directories       don't create host directories
--cut-dirs=NUMBER                ignore NUMBER remote directory components
-r,  --recursive                 specify recursive download
-l,  --level=NUMBER              maximum recursion depth (inf or 0 for infinite)
-P,  --directory-prefix=PREFIX   save files to PREFIX/..

June 26, 2020

#Check a site's downtime with this bash one-liner

server-side

During a site’s downtime, the following line will print the current time and try to access the given URL one time every 2 seconds. Only the received HTTP headers will be shown.

You may find it useful for monitoring downtime while you try to solve a problem.

while true; do date; curl -IX GET '<URL>'; sleep 2; done;

March 12, 2020

#List running queries in a MariaDB server

mariadb server-side

mysql -u$MYSQL_USER -p$MYSQL_PASSWORD $MYSQL_DATABASE
SHOW FULL PROCESSLIST;
## OR
mysqladmin -u$MYSQL_USER -p$MYSQL_PASSWORD processlist

March 10, 2020

#How to dump and restore PostgreSQL databases

postgresql server-side docker kubernetes

Options for dumping data:

# Binary mode
pg_dump -Fc -U $POSTGRES_USER $POSTGRES_DB > dump-`date '+%Y_%m_%d__%H_%M_%S'`.psqlc

# SQL mode
pg_dump -U $POSTGRES_USER $POSTGRES_DB > dump-`date '+%Y_%m_%d__%H_%M_%S'`.sql

# Docker and SQL mode
docker exec -it -u postgres <container_name> sh -c 'pg_dump -U $POSTGRES_USER $POSTGRES_DB' > dump-`date '+%Y_%m_%d__%H_%M_%S'`.sql

# Kubernetes and SQL mode (vulnerable to networking failures)
kubectl -n <namespace> exec deploy/postgres -- sh -c 'pg_dump -U $POSTGRES_USER $POSTGRES_DB' > dump-`date '+%Y_%m_%d__%H_%M_%S'`.sql

Options for restoring data:

# Binary mode
pg_restore -O -U $POSTGRES_USER -c -x -n public -d $POSTGRES_DB dump.psqlc

# SQL mode
psql -d $POSTGRES_DB -a -f /backups/dump.sql

If you’re using Docker, you can also place dump files on /docker-entrypoint-initdb.d and those will be imported on the first run. Accepted files types are *.sql, *.sql.gz, and *.sh.

March 05, 2020

#Find the biggest files and folders on a Linux server

server-side

du -ah / | sort -n -r | head -n 10

March 04, 2020

#How to manually create a Let's Encrypt certificate using Docker and the DNS challenge

server-side docker

docker run -it -v ${PWD}:/etc/letsencrypt --entrypoint="sh" certbot/certbot
certbot certonly -d <domain_name> --manual --preferred-challenges dns

March 02, 2020

#Format a new SSD/HD drive for usage

server-side

# Create the partition
parted --align optimal <raw_device>
mklabel msdos
mkpart primary ext4 0% 100%
quit

# Get the new partition device path
lsblk

# Make the filesystem
mkfs.ext4 <new_partition_device_path>

# Test the partition
mkdir /tmp/partition-test
mount -t ext4 <new_partition_device_path> /tmp/partition-test

# Get the UUID
blkid

# Edit the /etc/fstab to mount partition during boot
UUID=<uuid>     /storage        ext4    defaults,discard        0       2
## OR, if using XFS
UUID=<uuid>     /storage       xfs     defaults        1       2

March 01, 2020

#Basic shortcuts for navigating psql command line (PostgreSQL)

postgresql server-side

psql -U $POSTGRES_USER $POSTGRES_DB
\du  # list all users
\l # list databases
\c # select a database
\d # list tables
\d+ <table_name> # describe table

July 01, 2019

#Dump and restore data from MongoDB in Docker

mongodb docker server-side

# Create the database dump on your server
docker exec <my_mongodb_container> mongodump --archive=/backups/mongodb-`date +%Y%m%d`.gz --gzip --db <database_name>

# Copy it to your local machine, if needed
scp -r <server_user>@<server_ip>:<remote_backup_path> <local_backup_path>

# Restore it directly into a running Docker container
zcat <backup_path> | docker exec -i <my_mongodb_container> mongorestore --archive --drop

February 01, 2019

#Update URLs in a multisite Wordpress installation after a domain change

wordpress server-side

You will need wp-cli for this.

wp search-replace --url=<original_domain> <original_domain> <new_domain> --recurse-objects --network --skip-columns=guid
wp search-replace --network <original_domain> <new_domain>

If you are also transforming a subdomain installation into a directory based one, read through this and this.

September 01, 2018

#Sane defaults for our most common rsync needs

server-side

rsync -avhP -e 'ssh -p <optional_server_port>' --info=progress2 <server_user>@<server_up>:<path_to_be_copied> <destination_path>

<laury.dev/>

#Snippets

#Enable Intel integrated GPUs on Linux systems

#Troubleshoot CORS related issues on API requests

Kubernetes users running Ingress Nginx

Further reading

#Use iptables to allow only Cloudflare HTTP requests to target your server

Further reading

#Basic survival guide and commands for iptables

References

#Protect public URLs with ModSecurity for Nginx on a Kubernetes cluster

Customize ModSecurity rules on a per-host basis

How to completely customize modsecurity.conf

Further reading

#Host publicly accessible web URLs from a home server using a remote proxy

#Issue Let's Encrypt certificates for domains in a Kubernetes cluster

Further reading

#Pretty-print JSON files using the command line

#Quick usage examples of "find" on the command line

#Read Kubernetes API data using Golang

Further reading

#Read Kubernetes metrics-server data using Golang

#Receive data from Kubernetes API using curl

Rancher users

Further reading

#Activate the replicaset mode in MongoDB

#List Kernel modules installed in a Linux machine

#Test networking performance between two Linux machines

#Wipe any filesystem data and partitions from devices on Linux

#Download files recursively through a FTP connection using wget

#Check a site's downtime with this bash one-liner

#List running queries in a MariaDB server

#How to dump and restore PostgreSQL databases

#Find the biggest files and folders on a Linux server

#How to manually create a Let's Encrypt certificate using Docker and the DNS challenge

#Format a new SSD/HD drive for usage

#Basic shortcuts for navigating psql command line (PostgreSQL)

#Dump and restore data from MongoDB in Docker

#Update URLs in a multisite Wordpress installation after a domain change

#Sane defaults for our most common rsync needs