Inspecting Kubernetes Deployment Logs with kail

When systems start behaving strangely, inspecting live data becomes a vital part of managing operations of any scale. In addition to actual metrics like request throughput, database operations, and resource usage, real-time service logs are a fundamental piece of information about the health of your infrastructure.

In hosted environments like public clouds, it's often possible to access service logs using hosted solutions like Stackdriver on Google Cloud. If you've adopted Kubernetes to orchestrate your containers, though, you're also able to manually check the flood of raw Kubernetes logs.

Using the built-in kubectl command logs would be a straightforward way, but sadly it's not as flexible to use when you, for example, want to watch multiple pods.

In this case, my tool of choice is kail. It's amazingly simple to use and more flexible than any other workaround I've tried with the default Kubernetes utilities.

Let's say we deploy a self-hosted Docker registry using Helm and we want to stream the real-time registry pod logs to our terminal. The deployment command and output is shown below for the sake of completeness:

$ helm install \
  --name registry \
  --set-string replicaCount=3 \
  stable/docker-registry
NAME:   registry
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                             DATA  AGE
registry-docker-registry-config  1     0s

==> v1/Pod(related)
NAME                                       READY  STATUS
registry-docker-registry-56555c5b54-gnrn7  0/1    ContainerCreating
registry-docker-registry-56555c5b54-nq9b6  0/1    ContainerCreating
registry-docker-registry-56555c5b54-xqc9s  0/1    ContainerCreating

==> v1/Secret
NAME                             TYPE    DATA  AGE
registry-docker-registry-secret  Opaque  1     0s

==> v1/Service
NAME                      TYPE       CLUSTER-IP     EXTERNAL-IP
registry-docker-registry  ClusterIP  10.103.140.84  <none>

==> v1beta1/Deployment
NAME                      READY  UP-TO-DATE  AVAILABLE  AGE
registry-docker-registry  0/3    3           0          0s

And to get a feeling of how simple it is to get insights into our logs, run

$ kail -l app=docker-registry
default/registry-xqc9s[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-xqc9s[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-gnrn7[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-nq9b6[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-nq9b6[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-gnrn7[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-xqc9s[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"

That's mostly it! Here's a few additional options you might want to use with kail:

# View logs of resources with label
kail --label <name>=<value>

# View logs of pod
kail --pod <pod>

# View logs of service
kail --svc <service>

# View logs of deployment
kail --deploy <deployment>

# View older logs, for example since `10m`, `1h` or `3h15m`
kail --since <time>

Please go ahead and try it out for yourself or head over to the repository 🚀 I hope you enjoyed this showcase of yet another Kubernetes tool I use pretty much on a daily basis. If you've got any feedback, suggestions or questions, don't hesitate to slide into my DMs or send a mail!