Aug 11, 2019 · ⏱ 1 minute

Inspecting Kubernetes Deployment Logs with kail 🕵🏼‍♂️

When systems start behaving strangely, inspecting live data becomes a vital part of managing operations of any scale. In addition to actual metrics like request throughput, database operations, and resource usage, real-time service logs are a fundamental piece of information about the health of your infrastructure.

In hosted environments like public clouds, it's often possible to access service logs using hosted solutions like Stackdriver on Google Cloud. If you've adopted Kubernetes to orchestrate your containers, though, you're also able to manually check the flood of raw Kubernetes logs.

Using the built-in kubectl command logs would be a straightforward way, but sadly it's not as flexible to use when you, for example, want to watch multiple pods.

In this case, my tool of choice is kail. It's amazingly simple to use and more flexible than any other workaround I've tried with the default Kubernetes utilities.

Let's say we deploy a self-hosted Docker registry using Helm and we want to stream the real-time registry pod logs to our terminal. The deployment command and output is shown below for the sake of completeness:

$ helm install \
--name registry \
--set-string replicaCount=3 \
stable/docker-registry
NAME: registry
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
registry-docker-registry-config 1 0s
==> v1/Pod(related)
NAME READY STATUS
registry-docker-registry-56555c5b54-gnrn7 0/1 ContainerCreating
registry-docker-registry-56555c5b54-nq9b6 0/1 ContainerCreating
registry-docker-registry-56555c5b54-xqc9s 0/1 ContainerCreating
==> v1/Secret
NAME TYPE DATA AGE
registry-docker-registry-secret Opaque 1 0s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP
registry-docker-registry ClusterIP 10.103.140.84 <none>
==> v1beta1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
registry-docker-registry 0/3 3 0 0s

And to get a feeling of how simple it is to get insights into our logs, run

$ kail -l app=docker-registry
default/registry-xqc9s[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-xqc9s[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-gnrn7[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-nq9b6[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-nq9b6[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-gnrn7[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"
default/registry-xqc9s[registry]: 10.1.0.1 - "GET / HTTP/1.1" 200 0 "" "kube-probe/1.14"

That's mostly it! Here's a few additional options you might want to use with kail:

# View logs of resources with label
kail --label <name>=<value>
# View logs of pod
kail --pod <pod>
# View logs of service
kail --svc <service>
# View logs of deployment
kail --deploy <deployment>
# View older logs, for example since `10m`, `1h` or `3h15m`
kail --since <time>

Please go ahead and try it out for yourself or head over to the repository 🚀 I hope you enjoyed this showcase of yet another Kubernetes tool I use pretty much on a daily basis. If you've got any feedback, suggestions or questions, don't hesitate to slide into my DMs or send a mail!