<laury.dev/>

#Snippets

Topics
>
server-side (27)
>
docker (14)
>
kubernetes (14)
>
development (8)
>
homelab (8)
>
raspberry-pi (4)
>
golang (3)
>
survival-guide (3)
>
macos (2)
>
mariadb (2)
>
mongodb (2)
>
postgresql (2)
October 23, 2022

#Create a Docker image from scratch using an OS as base

docker

For the purposes of this example, we’ll be using Raspberry Pi OS 64-bit as the base filesystem for our new Docker image. Since this operating system lacks an official one on Docker Hub, it feels like a good fit. However, the principles described here should work well enough for any OS.

Choose an image to download here and save it locally. With unxz, uncompress the file.

unxz 2022-04-04-raspios-bullseye-arm64-lite.img.xz

Since Raspberry Pi OS is distributed as a disk image and not a partition image, we need to mount it with an offset to avoid getting data from the boot partition. Keep in mind that our goal here is to create a Docker image and, for that, we don’t need boot files.

Find the desired partition offset using fdisk.

fdisk -l 2022-04-04-raspios-bullseye-arm64-lite.img

You’re likely to see an output similar to the following:

Disk 2022-04-04-raspios-bullseye-arm64-lite.img: 1,86 GiB, 2000683008 bytes, 3907584 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0ee3e8a8

Device                                      Boot  Start     End Sectors  Size Id Type
2022-04-04-raspios-bullseye-arm64-lite.img1        8192  532479  524288  256M  c W95 FAT32 (LBA)
2022-04-04-raspios-bullseye-arm64-lite.img2      532480 3907583 3375104  1,6G 83 Linux

From this, we can see that the block size used is 512 and the start block for the root partition is 532480. Therefore, our offset will be 532480 * 512 = 272629760.

Now, let’s mount the relevant partition, gather its contents as a tar file, generate the image and push it to a registry. 

mkdir rootfs

mount -o loop,offset=272629760 2022-04-04-raspios-bullseye-arm64-lite.img rootfs

# Keep in mind that a container image is, under the hood, a tar file of tar files
tar -C rootfs -c . | docker import --platform=linux/arm64 - laury/raspberry-pi-os:bullseye-20220404
docker push laury/raspberry-pi-os:bullseye-20220404

umount rootfs

Note: as the time of writing, the --platform flag is ignored by docker import (tested on version 20.10). So, the host architecture is used to determine the platform supported by the generated image. This bug will be resolved on the 22.06 release (see this PR for more information).

Section break
October 03, 2022

#Use Lima as a Docker Desktop replacement on MacOS

dockerdevelopmentmacos

Lima (short for Linux on Mac) is a Virtual Machine management tool for running Linux boxes (think Vagrant, but for development tooling). It focuses on offering a convenient way to run and interact with containerd, but its high customizability allows for a lot more. The examples folder in its repository gives a good idea of its capabilities.

For running Lima as a Docker Desktop replacement, the official example provided is a great starting point with a few shortcomings:

  • mounting only a temporary folder (/tmp/lima) as writable
  • not including support for building and running multi-arch containers

To circumvent these issues, download this YAML and run:

limactl start docker.yml

If docker login gives you trouble, make sure that ~/.docker/config.json uses osxkeychain as the credsStore. Example:

{
  "auths": {},
  "credsStore": "osxkeychain"
}

Considerations about chown and chmod

Lima uses Reverse SSHFS by default to mount folders from host to guest. This strategy provides good compatibility but can cause problems in some cases. For instance, Containers that run chown on startup like Jupyter and Gogs might not be able to start up at all.

In some situations, you might need to experiment with 9p and reverse-sshfs as mount types.

Further reading

Filesystem mounts available for Lima

VM customization options

Section break
November 24, 2021

#Pull and push container images from and to AWS ECR repositories via Docker CLI

awsdocker

To use the regular Docker CLI with Amazon Elastic Container Registry (Amazon ECR), you have to acquire specific login credentials for it first. Assuming you already got basic AWS credentials, run the following to pass those on to Docker. Just note that, below, AWS_ACCOUNT_ID refers to the repository owner, which may not be your own AWS account.

aws ecr get-login-password --region <AWS_REGION> | docker login -u AWS --password-stdin https://<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com

# Example for the official Sagemaker ECR repository
aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin https://683313688378.dkr.ecr.us-east-1.amazonaws.com

Once you’re logged in, list all available container images in a repository with the following:

aws ecr describe-images --region <AWS_REGION> --registry-id <AWS_ACCOUNT_ID> --repository-name <REPOSITORY_NAME>

# Example for the official Sagemaker ECR repository
aws ecr describe-images --region us-east-1 --registry-id 683313688378 --repository-name sagemaker-scikit-learn

Further reading

Prebuilt Amazon SageMaker Docker Images for Scikit-learn and Spark ML

Section break
November 14, 2020

#Get a Docker container's IP from the command line

docker

When dealing with container IPs, it’s important to take note of which network each container is connected to and which network mode it’s using. The default network mode for a Docker installation is bridge, so I’ll assume you’re using it too.

Any container started with docker run will be attached to a default bridge network. To get its IP on that network, run:

docker inspect --format "{{.NetworkSettings.Networks.bridge.IPAddress}}" <container_name>

Note: the --format flag accepts Go’s template syntax. Take a look at some Docker usage examples.

For containers started with docker-compose, a new bridge network is created. Its default name will be <folder_name>_default, so the above command would look like this:

docker inspect --format "{{.NetworkSettings.Networks.<folder_name>_default.IPAddress}}" <container_name>

In both cases, the received IPs are reachable from the host and from other containers in the same bridge.

Further reading

Official docs on Docker bridge networks

Section break
November 13, 2020

#Simple Docker one-liner to run Jupyter Notebook with Python

developmentpythondocker

The following command will start a Jupyter instance mounting the current folder.

docker run --rm -v ${PWD}:/home/jovyan -p 8888:8888 -e JUPYTER_ENABLE_LAB=yes -e CHOWN_HOME=yes jupyter/scipy-notebook:latest

Explanation about the chosen environment variables:

JUPYTER_ENABLE_LAB=yes  # enable the superior "lab" interface
CHOWN_HOME=yes          # ensure the home directory is writable for Jupyter

Further reading

List of all environment variables available using Docker

Official documentation about Jupyter Docker stacks

Section break
September 09, 2020

#Install Kubernetes on Raspberry Pi OS

raspberry-pikubernetesdockerhomelab

We will use k3s, a lightweight Kubernetes distribution, to get the most out of our hardware. This tutorial uses a Raspberry Pi 4 and the latest Raspberry Pi 32-bit (formerly known as Raspbian). The 64-bit version is pretty much the same for this step-by-step.

Flash the OS image on your SD Card and, if necessary, add Wi-Fi credentials so you can access it.

Enable cgroups support and disable IPv6 by appending the following on /boot/cmdline.txt (remember that /boot refers to the boot partition on your SD Card).

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory ipv6.disable=1

I also recommend disabling swap.

dphys-swapfile swapoff && systemctl disable dphys-swapfile.service

If your workloads don’t require GPU, you may want to change the Memory Split to 16 using raspi-config. You’ll have a little extra RAM this way.

You can force the OS to use legacy iptables to ensure compatibility with older Kubernetes versions, but this step is optional.

iptables -F
update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Install and test k3s.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -

# After a minute, you should be able to test it
kubectl get nodes

The kubeconfig yaml will be available at /etc/rancher/k3s/k3s.yaml.

If you have worker nodes to add, use the following command. Find the cluster token at /var/lib/rancher/k3s/server/node-token on the server node.

curl -sfL https://get.k3s.io | K3S_URL=https://<server_ip>:6443 K3S_TOKEN="<cluster_token>" sh -

Maintenance

Things should run smoothly from here, but if they don’t, the next few commands might help:

# See current status of k3s
systemctl status k3s

# See system logs (including k3s)
journalctl -xe

# Uninstall k3s
/usr/local/bin/k3s-uninstall.sh

Kubernetes automatically performs cleanup of unused containers and images on nodes. Still, if you’re under the impression that your filesystem is getting bloated by these, you can force a safe deletion with:

k3s crictl rmi --prune

Further reading

Configuration options for k3s

Section break
September 08, 2020

#Use the RGB Cooling HAT, by Yahboom, with Docker on your Raspberry Pi

raspberry-pidockerhomelab

This tutorial was tested on a Raspberry Pi 4 with RGB Cooling HAT model “YB-EBV02 VER1.1”, by Yahboom, on 32-bit and 64-bit OS.

The hardware creators do provide installation documentation and official code, but putting all parts together can be challenging.

If you’re willing to use Docker and run the 32-bit version of Raspberry Pi OS, you can activate everything with one command (after enabling I2C with raspi-config).

docker run -d --restart unless-stopped --network host --privileged laury/raspberry-pi-rgb-cooling-hat:latest

To run the software on a 64-bit OS, you’ll need to enable Qemu first and then run the container with the --platform flag.

docker run --privileged --rm tonistiigi/binfmt --install all

docker run -d --restart unless-stopped --platform="linux/arm/v7" --network host --privileged laury/raspberry-pi-rgb-cooling-hat:latest

The container image tagged with latest enables the default functionality of the accessory. If you’d like for the RGBs to the turned off, use the v1.1.0-noRGB image tag.

You can also build the Docker image yourself using the following Dockerfile. Just remember to target ARMv7 architecture. The easiest way to achieve that is to build in the Raspberry Pi itself.

FROM python:2.7

RUN apt update &&\
    apt install -y i2c-tools git

RUN pip install Adafruit-GPIO==1.0.3 \
                Adafruit-BBIO==1.2.0 \
                Adafruit-SSD1306==1.6.2 \
                smbus==1.1.post2 \
                image==1.5.32 \
                vcgencmd==0.1.1 \
                RPi.GPIO==0.7.1

WORKDIR /tmp
RUN git clone --depth 1 https://github.com/raspberrypi/firmware.git &&\
    cp -a firmware/hardfp/opt/vc/* /usr

RUN git clone https://github.com/YahboomTechnology/Raspberry-Pi-RGB-Cooling-HAT.git &&\
    unzip Raspberry-Pi-RGB-Cooling-HAT/4.Python\ programming/RGB_Cooling_HAT.zip &&\
    mkdir /app &&\
    cp -a RGB_Cooling_HAT/* /app &&\
    rm -rf /tmp/*

WORKDIR /app
CMD python /app/RGB_Cooling_HAT.py

Section break
August 23, 2020

#Create, test and run a Java project with Gradle using Docker and no IDE

javadevelopmentdocker

In order to create and manage Java projects, its common to employ the help of an IDE like IntelliJ IDEA or Eclipse.

However, if you would rather use only the command line, Docker can be a great help.

For convenience and reproducibility, create a Dockerfile in your local filesystem with the following contents.

FROM openjdk:14

ENV GRADLE_HOME=/opt/gradle/gradle-6.4.1
ENV PATH=${GRADLE_HOME}/bin:${PATH}

RUN yum install -y wget unzip

RUN wget https://services.gradle.org/distributions/gradle-6.4.1-bin.zip -P /tmp &&\
    unzip -d /opt/gradle /tmp/gradle-*.zip &&\
    rm -rf /tmp/*

Build the Docker image and run a container (takes a while). Again, for convenience, -v and -w are used so that file creations and modifications made inside the container are reflected outside.

docker run --rm -it -v ${PWD}:/app -w /app $(docker build -q .) bash

Now, we can use gradle to create and manage a project from the command line.

# Create a new project
# The "application" template most likely will suit your needs
gradle init

# Run unit tests (if you enabled JUnit during project creation)
gradle test

# Run your code
gradle run

Write source code in src/main/java/<your_package> and test code in src/test/java/<your_package>.

Further reading

Official Gradle guide for Java applications

Docker command line

Gradle command line

List of JUnit Assertions

Section break
March 13, 2020

#How to dump and restore MySQL/MariaDB databases

mariadbkubernetesdocker

Options for dumping data:

# Directly
mysqldump --databases $MYSQL_DATABASE -u$MYSQL_USER -p$MYSQL_PASSWORD > dump-`date '+%Y_%m_%d__%H_%M_%S'`.sql

# Docker
docker exec -it <container_name> sh -c 'mysqldump --databases $MYSQL_DATABASE -u$MYSQL_USER -p$MYSQL_PASSWORD' > dump-`date '+%Y_%m_%d__%H_%M_%S'`.sql

# Kubernetes (vulnerable to networking failures)
kubectl -n <namespace> exec deploy/<deploy_name> -- bash -c 'mysqldump --databases $MYSQL_DATABASE -u$MYSQL_USER -p$MYSQL_PASSWORD' > dump-`date '+%Y_%m_%d__%H_%M_%S'`.sql

Additionally, if your database operates in a remote server (like Amazon RDS), you may still use a local Docker container for dumping the data.

# Create a local container with the desired version (in this example, MariaDB 10.4 is used)
docker run -it --rm -v ${PWD}:/dump -w /dump mariadb:10.4 bash

# Get the data
mysqldump -h <hostname> --databases <database_name> -u <database_user> --password='<password>' > dump-`date '+%Y_%m_%d__%H_%M_%S'`.sql

For restoring data, you can use:

mysql -u$MYSQL_USER -p$MYSQL_PASSWORD < dump.sql

If you’re using Docker, you can also place dump files on /docker-entrypoint-initdb.d and those will be imported on the first run. Accepted files types are *.sql, *.sql.gz, and *.sh.

Section break
March 11, 2020

#How to publish Ionic apps on the Google Play Store

ionicandroiddocker

We’ll use Docker to avoid installing the Android SDK on our computer for this task. This is possible thanks to this ionic Docker image.

# Enter in the container command line
docker run -it --rm --net host --privileged -v /dev/bus/usb:/dev/bus/usb -v ~/.gradle:/root/.gradle -v $PWD:/Sources:rw -v /home/<your_user>/.ssh:/root/.ssh hacklab/ionic:android-28 bash

# Install javascript dependencies
npm install

# Execute Ionic build
ionic cordova platform add android
ionic cordova build android --prod --release

# Sign the generated build
jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore ~/.ssh/<your_keystore> platforms/android/app/build/outputs/apk/release/app-release-unsigned.apk myapp
/opt/android-sdk-linux/build-tools/28.0.3/zipalign -v 4 platforms/android/app/build/outputs/apk/release/app-release-unsigned.apk myapp-2.0.0.apk

Section break
March 10, 2020

#How to dump and restore PostgreSQL databases

postgresqlserver-sidedockerkubernetes

Options for dumping data:

# Binary mode
pg_dump -Fc -U $POSTGRES_USER $POSTGRES_DB > dump-`date '+%Y_%m_%d__%H_%M_%S'`.psqlc

# SQL mode
pg_dump -U $POSTGRES_USER $POSTGRES_DB > dump-`date '+%Y_%m_%d__%H_%M_%S'`.sql

# Docker and SQL mode
docker exec -it -u postgres <container_name> sh -c 'pg_dump -U $POSTGRES_USER $POSTGRES_DB' > dump-`date '+%Y_%m_%d__%H_%M_%S'`.sql

# Kubernetes and SQL mode (vulnerable to networking failures)
kubectl -n <namespace> exec deploy/postgres -- sh -c 'pg_dump -U $POSTGRES_USER $POSTGRES_DB' > dump-`date '+%Y_%m_%d__%H_%M_%S'`.sql

Options for restoring data:

# Binary mode
pg_restore -O -U $POSTGRES_USER -c -x -n public -d $POSTGRES_DB dump.psqlc

# SQL mode
psql -d $POSTGRES_DB -a -f /backups/dump.sql

If you’re using Docker, you can also place dump files on /docker-entrypoint-initdb.d and those will be imported on the first run. Accepted files types are *.sql, *.sql.gz, and *.sh.

Section break
Section break
July 01, 2019

#Dump and restore data from MongoDB in Docker

mongodbdockerserver-side

# Create the database dump on your server
docker exec <my_mongodb_container> mongodump --archive=/backups/mongodb-`date +%Y%m%d`.gz --gzip --db <database_name>

# Copy it to your local machine, if needed
scp -r <server_user>@<server_ip>:<remote_backup_path> <local_backup_path>

# Restore it directly into a running Docker container
zcat <backup_path> | docker exec -i <my_mongodb_container> mongorestore --archive --drop

Section break