In Part 1 of this series I touched on containers, Docker, and how these technologies are rapidly redefining operations and infrastructure across the industry. Part 2 continued the discussion by going over Kubernetes, what it is and what it provides. Now with Part 3 of this series I’ll be going over how to get started with Kubernetes and providing some recommendations on how to structure your work.

Table of Contents

This is a long post with a lot of technical details and example files. As such, here’s a TOC to provide easy access to the sections you are interested in.

• Commands
• Local Development
• Services
• Containers
• Secrets

Commands

Managing a Kubernetes cluster is done via the kubectl command line utility. If you’re using Google Cloud, a build of kubectl is provided by the SDK automatically. While it is possible to fully configure and command a cluster using just the kubectl command line tool, I highly recommend writing out service configuration files and applying them to the cluster. This way you have files you can version control that are the canonical source of your configuration. More on this later.

kubectl contains a myriad of commands for managing your cluster as well as providing insight into the current status of the cluster and it’s components. Here are the commands I’ve found myself using on a regular basis.

Management

kubectl apply -f service-file.yml

The apply command is a catch-all for applying changes to the cluster from a file. Given a configuration file, Kubernetes will figure out the changes between what’s running and what is in the passed in file, making any changes necessary to update the live system.

kubectl rollout

Any change triggered with apply or a few other commands will trigger a new Rollout. Use kubectl rollout status to check the status of the latest requested change, cancel the rollout or roll back to a previous version of the resource.

Introspection

kubectl describe [pod,service,...] [resource]

This command provides detailed information about the requested resource. When something is going wrong, your first step should be to check out what describe says about the problem resource.

kubectl logs [resource]

Containers by default are expected to send all log output to STDOUT, to be vacuumed up by Kubernetes. This command gives you access to those log entries and will show you the most recent lines received for the given Pod / container. To get a live view of the log, use kubectl logs -f [resource] (follow) to the command.

kubectl get [pods,deployments,services,...]

List out all current running resources in the default namespace. To see resources in a specific namespace use --namespace=[my namespace] and to see resources across all namespaces use --all-namespaces.

kubectl exec

This command is a wrapper around docker exec, letting you execute commands on a container or even on a Pod if that Pod only is running one container. I commonly use this to get a bash prompt on a running pod, which looks like: kubectl exec -it [pod name] -- /bin/bash. The -i hooks up STDIN and -t turns STDIN into a TTY so we get a fully functional bash prompt. If a Pod is running multiple containers, you can choose the specific container to jump into with -c [container-name].

Deploying New Images

Currently, there is no command available to tell a Deployment to deploy new Pods with the newest version of their configured container. Given that deploying new code is the most common operation that will be requested of any Kubernetes cluster, this omission requires some decisions to figure out how you want to deploy new containers. I evaluated a few ideas before settling on one:

• Tag containers to :latest or something similar and manually delete pods, letting the Deployment bring new ones up.
• Update service configuration files to point to every new container label e.g. redis-cache:9c713a and apply that file to the cluster. This requires a new commit to these files for every deploy.
• Manually tell the Deployment’s Pods to change their image: kubectl set image deployment/[service name] *=[new image].

I wanted a solution that was easily scriptable that would leave a historical trail inside of Kubernetes itself. I also didn’t want to get bogged down in constantly committing new files and polluting my repository’s history with deployment records, so I settled on the third option. Specifically, my deployment script runs the following:

• All Pod templates reference the :latest tag of their container
• New container builds get pushed to :latest as well as a label named after the HEAD git commit hash (e.g. :9c713a)
• Tell the Deployment in question to use the new container with the git commit hash tag (kubectl set image).

With these rules I get a few automatic benefits. First, setting the image on all Pods with set image will gracefully roll out the change across all of a Deployment’s Pods, which you can watch with kubectl rollout status. Second, if a Pod gets killed or new Pods come online for whatever reason, they will always automatically get the latest version of the code. Third, I can use the same docker registry for minikube usage without worry of breaking production pods; I leave the :latest label alone until I’m ready to send code to production.

For more information about handling containers, please see the Containers section further down.

Local Development

Following Kubernete’s comprehensive and incredible documentation, the team has also provided a way to easily run your own cluster locally with minikube! Minikube works with multiple virtualization platforms to set up a single node, fully functional Kubernetes cluster. Once the cluster is running, minikube provides plenty of tools for access and introspection of your cluster, as well as configuring kubectl automatically. The most common tools you’ll use are:

minikube service [service name] --url

This will print out one or many URLs (depending on the Service configuration) to the given Service’s endpoint running on your local machine.

minikube dashboard

This will start up the Kubernetes dashboard, giving you a web-based look into how the cluster is running.

One issue I ran into pretty quickly was that my default VM settings were too low (1 CPU and 1GB of RAM). I recommend bumping those up a decent bit before starting up minikube. For example to configure 4 CPUs and 8GB of RAM and run the cluster with VMWare Fusion:

minikube config set cpus 4
minikube config set memory 8192
minikube start --vm-driver vmwarefusion

To leave minikube you’ll need to reconfigure kubectl to talk to a different cluster context:

kubectl config get-contexts
kubectl config use-context [cluster context name from above]

Likewise you can manually re-point kubectl to minikube with kubectl config use-context minikube.

Service configuration

Service configuration files can be written in JSON or YAML. I prefer to use YAML as I find YAML easier to write and easier to read. YAML also supports comments which can be invaluable in more complex infrastructures.

For most configurations, Services will be the highest levels of infrastructure that you’ll need to configure. As such Service-first is how I’ve decided to structure my files. Each Service gets a directory which will contain all files required for that Service to run. To help with searchability, I have one required file in each Service directory, a [service name]-k8s.yml file which contains all of the Kuberentes-specific configuration for that Service.

For a visual example, here’s what the my Redis caching Service looks like:

services/
    redis-cache/
        Dockerfile
        redis.conf
        redis-cache-k8s.yml

And here’s the contents of redis-cache-k8s.yml.

apiVersion: v1
kind: Service
metadata:
  name: redis-cache
  labels:
    role: cache
spec:
  type: NodePort
  ports:
  - port: 6379
    targetPort: 6379
  selector:
    role: cache

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: redis-cache
spec:
  # https://github.com/kubernetes/kubernetes/issues/23597
  revisionHistoryLimit: 3
  replicas: 1
  template:
    metadata:
      labels:
        role: cache
    spec:
      containers:
      - name: redis
        image: redis:3
        resources:
          requests:
            cpu: 100m
            memory: 1Gi
        ports:
          - containerPort: 6379
 

With this setup, starting up the Service or making any changes to the Service or its Deployment is a simple kubectl apply -f service/redis-cache/redis-cache-k8s.yml away.

Containers and Registries

Tagging Containers

I currently tag every production container build with two tags: :latest and the git commit hash of HEAD, e.g. :9c713a. There are a lot of people who strongly recommend not using :latest but that is mainly relevant if you’re only using :latest. Please see Deploying New Images above for my full rationale behind this tagging decision.

Minikube Access

One issue I ran into early on with minikube was ensuring Pods had permission to pull Docker images from my Google account’s private registry. When running Kubernetes on GKE itself, all servers are automatically seeded with permissions to access this registry, but locally minikube does not have these permissions.

The solution is to use the imagePullSecrets value in your Pod spec. To do this on Google Cloud, go to IAM and create a new Service Account there with the permissions Storage -> Storage Object Viewer. Make sure to specify “Furnish a new private key”. This will give you a JSON file with that user’s credentials that you’ll need to store locally. With this information in hand, I then have a shell script that creates a new Secret.

#!/usr/bin/env sh

SPATH="$(cd $(dirname "$0") && pwd -P)"
SECRET_NAME=${1:-docker-registry-secret}
CONFIG_PATH=${2:-$SPATH/localkube.json}
if [[ ! -f $CONFIG_PATH ]]; then
  echo "Unable to locate service account config JSON: $CONFIG_PATH";
  exit 1;
fi

kubectl create secret docker-registry $SECRET_NAME  \
  --docker-server "https://gcr.io" \
  --docker-username _json_key \
  --docker-email [service account email address] \
  --docker-password="`cat $CONFIG_PATH`" ${@:3}
	

This script creates a Secret named docker-registry-secret which can then be referenced in your Service config. Make sure you’re also referencing the full path to your container and you should be good to go!

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: redis-cache
spec:
  ...
  template:
    spec:
      imagePullSecrets:
      - name: docker-registry-secret
      ...
      containers:
      - name: redis
        image: gcr.io/[google account id]/redis-cache:latest
  ...

Secrets

Secret values like passwords, certificates, and keys, always require careful handling and are easy to get wrong. On one hand, you don’t ever want to commit plain secrets to a source repository, no matter how privately hosted that repository is. On the other hand you want the values stored somewhere for a canonical source, preferably source control for change tracking. Having tried multiple different encryption strategies in the past, today I recommend StackExchange’s BlackBox.

BlackBox uses PGP/GPG keys to encrypt files such that only specific users are allowed access. Adding a user requires that user’s public GPG key, and removing said user consists of removing their name from a list in a file. You then tell BlackBox which files you need encrypted and BlackBox does the rest, letting you safely commit encrypted secrets to source control.

Getting these secrets into Kubernetes requires some local scripting, as Kubernetes stores secrets in etcd in an unencrypted format. The structure of these secrets is completely up to you, but to help make some decisions here’s how my application is structured.

I currently keep two kinds of secrets: YAML documents with lots of key-value secrets (e.g. database and external service credentials) and single encrypted files (like SSL certificates and keys). I then have a rake task that will decrypt these files, build up the appropriate YAML document for a Kubernetes Secret and apply that file with kubectl apply -f - (the - means read from STDIN).

For example, if I have an encrypted YAML file with the content:

---
rails:
  secret_key_base: "..."
  service_api_key: "..."
database:
  username: "..."
  password: "..."

I can load this into a Kubernetes secret with the following code:

raw_secrets = `blackbox_cat secrets/my-secrets.yml.gpg`

secrets = YAML.load(raw_secrets)

secrets.each do |name, values|
  k8s_secret = {
    "apiVersion" => "v1",
    "kind" => "Secret",
    "type" => "Opaque",
    "metadata" => { "name" => name },
    "data" => {},
  }

  values.each do |key, value|
    k8s_secret["data"][key] = Base64.strict_encode64(value)
  end

  stdout, status = Open3.capture2("kubectl apply -f -", stdin_data: k8s_secret.to_yaml)
end

Then my Service configuration can reference these secrets by name (I use environment variables for this kind of Secret):

...
env:
  - name: RAILS_SECRET_KEY_BASE
    valueFrom:
      secretKeyRef:
        name: rails
        key: secret_key_base
  - name: RAILS_SERVICE_API_KEY
    valueFrom:
      secretKeyRef:
        name: rails
        key: service_api_key
  - name: DATABASE_USERNAME
    valueFrom:
      secretKeyRef:
        name: database
        key: username
  - name: DATABASE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: database
        key: password
	

I think that’s the majority of issues I ran into and decisions I had to make when figuring out this whole Kubernetes thing. Please throw any questions or comments you have at me in the Comments section below!

---

To skip around to other parts of this blog series, use the links below.

Part 1 - Looking at containerized infrastructure

Part 2 - What is Kubernetes and how does it make containerized infrastructure easy?