#Snippets

August 03, 2020

#Quick usage examples of "find" on the command line

Search recursively for files using a regex pattern. Use -name for a case sensitive search.

find . -iname '<partial_name>*'

Execute a command with every result found.

find . -iname '*.md' -type f -exec ls -lh {} \;

Note: every command needs to be terminated by ; or +. But these signs may need to be escaped as in ';' or \;. Outputs given by find will be placed on {}. Multiple uses of -exec is allowed.

Section break
August 02, 2020

#Read Kubernetes API data using Golang

Get the API client.

import (
  "fmt"
  "context"
  "k8s.io/client-go/kubernetes"
  "k8s.io/client-go/tools/clientcmd"
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// use the current context in kubeconfig
config, err := clientcmd.BuildConfigFromFlags("", "<path_to_KUBECONFIG>")
if err != nil {
  panic(err.Error())
}

// create the clientset
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
  panic(err.Error())
}

Usage examples:

  • read servers data
nodeList, err := clientset.CoreV1().Nodes().List(context.TODO(), metav1.ListOptions{})
if err != nil {
  panic(err.Error())
}
  • read pods data
namespace := "example"  // leave empty to get data from all namespaces
podList, err := clientset.CoreV1().Pods(namespace).List(context.TODO(), metav1.ListOptions{})
if err != nil {
  panic(err.Error())
}

ListOptions struct can take two important strings, field selectors and label selectors. This can be used

options := metav1.ListOptions{
  LabelSelector: "labelName=labelKey",
  FieldSelector: "spec.nodeName=<node_name>",  // Example for filtering by node name
}

Field selectors and label selectors on this case operate in the same way that the CLI options for kubectl do. So, the same rules apply here.

Further reading

Official docs on Field Selectors and Labels.

Section break
August 02, 2020

#Read Kubernetes metrics-server data using Golang

First, get the metrics API client.

import (
  "fmt"
  "context"
  "k8s.io/client-go/kubernetes"
  "k8s.io/client-go/tools/clientcmd"
  metricsv "k8s.io/metrics/pkg/client/clientset/versioned"
)

// Get the config
config, err := clientcmd.BuildConfigFromFlags("", "<path_to_KUBECONFIG>")
if err != nil {
  panic(err.Error())
}

// Get the metrics client
metricsClientset, err := metricsv.NewForConfig(config)
if err != nil {
  panic(err.Error())
}

Get the data and store it in podMetricsList array.

namespace := "example"  // leave empty to get data from all namespaces
podMetricsList, err := metricsClientset.MetricsV1beta1().PodMetricses(namespace).List(context.TODO(), metav1.ListOptions{})
if err != nil {
  panic(err.Error())
}

Iterate over the results found.

for _, v := range podMetricsList.Items {
  fmt.Printf("%s\n", v.GetName())
  fmt.Printf("%s\n", v.GetNamespace())
  fmt.Printf("%vm\n", v.Containers[0].Usage.Cpu().MilliValue())
  fmt.Printf("%vMi\n", v.Containers[0].Usage.Memory().Value()/(1024*1024))
}

Note: if you’re inspecting pods that may have more than one container, you’ll need to iterate over v.Containers as well.

Section break
August 01, 2020

#Receive data from Kubernetes API using curl

Start by using kubectl’s proxy to make the requests. This avoids having to deal with authentication headers.

kubectl proxy

Test the connection with a basic call.

curl http://localhost:8001/api/

To check the APIs groups available, access the root.

curl http://localhost:8001/apis/

Based on the groups list, you can create the URLs for next requests.

For cluster-scoped resources, use:

  • /apis/GROUP/VERSION/RESOURCETYPE
  • /apis/GROUP/VERSION/RESOURCETYPE/NAME

For namespace-scoped resources:

  • /apis/GROUP/VERSION/RESOURCETYPE
  • /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE
  • /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE/NAME

To get deployment data, the API Group is “apps”, Version is “v1” and Resourcetype is “deployments”. The final request looks like this:

curl http://localhost:8001/apis/apps/v1/deployments

If the Kubernetes cluster has a metrics-server, the following request will get node resource usage data:

curl http://localhost:8001/apis/metrics.k8s.io/v1beta1/nodes

Rancher users

However, if your cluster has been created using Rancher, you need to have “Authorized Cluster Endpoint” activated, so your requests can reach the API Server directly. If not, your URL paths will differ from the examples above.

On your command line, you will also have to select the correct context for kubectl before activating the proxy.

# List availabe contexts
kubectl config get-contexts

# Select the direct API context
# Its name has the format <cluster_name>-<pilot_name>
kubectl config use-context <direct_context>

Further reading

Overview of the Kubernetes API

More info about API concepts

Accessing the API without kubectl proxy

Full list of Groups, Versions and Resourcetypes

Section break
July 31, 2020

#Activate the replicaset mode in MongoDB

Start the MongoDB deamon with the --replSet flag.

mongod --replSet rs0

After first start, initialize the replica set with the following:

mongo <mongo_host>/<database> --eval "rs.initiate({_id: 'rs0', members: [ { _id: 0, host: '<mongo_host>:27017' } ]})"

Section break
Section break
July 23, 2020

#Test networking performance between two Linux machines

Let’s use iperf to measure networking performance. First, set a “server” machine.

iperf -s

From a “client” machine, run the test.

iperf -c <server_ip>

If you are having trouble getting the server’s IP, use ip addr show to find addresses used by the current machine in any connected networks.

Section break
July 23, 2020

#Wipe any filesystem data and partitions from devices on Linux

wipefs should work in most cases to clean filesystems.

# View filesystem data from device
wipefs /dev/<my_device>

# Wipe any filesystem found
wipefs --all /dev/<my_device>

If wipefs can’t detect a filesystem, you can quickly wipe the header data of the device with dd.

dd if=/dev/zero of=/dev/<my_device> bs=1M count=100 oflag=direct,dsync status=progress

You may need to restart your machine to make the filesystem deletion known to the OS, if your device is a “LVM2_member” for instance.

Also, as a last resort, you can completely wipe the device, if everything else failed.

dd if=/dev/zero of=/dev/<my_device> bs=1M count=5000000 oflag=direct,dsync status=progress

To clear partiptions, sgdisk will serve you well.

sgdisk --zap-all /dev/<my_device>

Section break
July 16, 2020

#Bootstrap a Go Cobra application with Go Modules

To bootstrap a Go application using Cobra Generator and Go Modules, start by installing the Cobra command line:

# Install Cobra
go get github.com/spf13/cobra/cobra

Initialize your Cobra project, but remember to use your package’s fully qualified name, as in: github.com/laurybueno/kubectl-hoggers. Don’t worry if your package is not publicly available yet. And there is no need to put your source code inside GOPATH anymore (since Go 1.11 at least):

cobra init --pkg-name <fully_qualified_package_name>

Initialize your new module:

go mod init <fully_qualified_package_name>

Get recursively all its dependencies and install it:

go get -d -v ./...
go install -v ./...

Your app’s command line is now ready to be used:

kubectl-hoggers

Further reading

Docker and Go: https://www.docker.com/blog/docker-golang/

Go Modules: https://github.com/golang/go/wiki/Modules

Section break
July 16, 2020

#Download files recursively through a FTP conection using wget

Unfortunately, SCP and Rsync aren’t always an option to get our files out of a server. When FTP is all we have, wget can still do a pretty good job and not re-download files you already have, like a primitive Rsync.

wget -nc -nH --cut-dirs=<REMOTE_FOLDERS_TO_CUT_FROM_PATH> --ftp-user='<FTP_USER>' --ftp-password='<FTP_PASSWORD>' -rl 0 -P <LOCAL_PATH> 'ftp://<REMOTE_HOST>/<PATH_TO_FILES>'

Explanation for the used flags:

-nc, --no-clobber                skip downloads that would download to existing files (overwriting them)
-nH, --no-host-directories       don't create host directories
--cut-dirs=NUMBER                ignore NUMBER remote directory components
-r,  --recursive                 specify recursive download
-l,  --level=NUMBER              maximum recursion depth (inf or 0 for infinite)
-P,  --directory-prefix=PREFIX   save files to PREFIX/..

Section break
June 26, 2020

#Check a site's downtime with this bash one-liner

During a site’s downtime, the following line will print the current time and try to access the given URL one time every 2 seconds. Only the received HTTP headers will be shown.

while true; do date; curl -IX GET '<URL>'; sleep 2; done;

Section break
Section break
Section break
Section break
March 11, 2020

#How to publish Ionic apps on the Google Play Store

For this task, we’re going to use Docker to avoid installing the Android SDK on our computer. Such magic is possible thanks to this ionic Docker image.

# Enter in the container command line
docker run -it --rm --net host --privileged -v /dev/bus/usb:/dev/bus/usb -v ~/.gradle:/root/.gradle -v $PWD:/Sources:rw -v /home/<your_user>/.ssh:/root/.ssh hacklab/ionic:android-28 bash

# Install javascript dependencies
npm install

# Execute Ionic build
ionic cordova platform add android
ionic cordova build android --prod --release

# Sign the generated build
jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore ~/.ssh/<your_keystore> platforms/android/app/build/outputs/apk/release/app-release-unsigned.apk myapp
/opt/android-sdk-linux/build-tools/28.0.3/zipalign -v 4 platforms/android/app/build/outputs/apk/release/app-release-unsigned.apk myapp-2.0.0.apk

Section break
March 10, 2020

#How to dump and restore PostgreSQL databases

# Binary mode
pg_dump -Fc -U $POSTGRES_USER $POSTGRES_DB > dump-`date '+%Y_%m_%d__%H_%M_%S'`.psqlc
pg_restore -O -U $POSTGRES_USER -c -x -n public -d $POSTGRES_DB dump-2019_03_18__19_03_14.psqlc

# SQL mode
pg_dump -U $POSTGRES_USER -f /backups/dump.sql $POSTGRES_DB
psql -d $POSTGRES_DB -a -f /backups/dump.sql

# Docker and SQL mode
docker exec -it -u postgres <container_name> sh -c 'pg_dump -U $POSTGRES_USER $POSTGRES_DB' > dump-`date '+%Y_%m_%d__%H_%M_%S'`.sql

Section break
Section break
Section break
March 02, 2020

#Format a new SSD/HD drive

# Create the partition
parted --align optimal <raw_device>
mklabel msdos
mkpart primary ext4 0% 100%
quit

# Get the new partition device path
lsblk

# Make the filesystem
mkfs.ext4 <new_partition_device_path>

# Test the partition
mkdir /tmp/partition-test
mount -t ext4 <new_partition_device_path> /tmp/partition-test

# Get the UUID
blkid

# Edit the /etc/fstab to mount partition during boot
UUID=<uuid>     /storage        ext4    defaults,discard        0       2
## OR, if using XFS
UUID=<uuid>     /storage       xfs     defaults        1       2

Section break
Section break
July 01, 2019

#Dump and restore data from MongoDB in Docker

# Create the database dump on your server
docker exec <my_mongodb_container> mongodump --archive=/backups/mongodb-`date +%Y%m%d`.gz --gzip --db <database_name>

# Copy it to your local machine, if needed
scp -r <server_user>@<server_ip>:<remote_backup_path> <local_backup_path>

# Restore it directly into a running Docker container
zcat <backup_path> | docker exec -i <my_mongodb_container> mongorestore --archive --drop

Section break
Section break
Section break
Section break
September 01, 2017

#Some useful Vagrant commands

# Bring up the system
vagrant up

# Get into the running virtual machine
vagrant ssh

# Bring the machine down and up again
# (equivalent of running a halt followed by an up)
vagrant reload

# Destroy the current created virtual machines
vagrant destroy