May 16, 2021

Improving observability with annotated build information

In my recent posts on managing a commit-oriented infrastructure, I explained the benefits of deploying services of one specific version. This version would be the hashed Git commit reference, or commit SHA.

When running this kind of setup, you might encounter the question of which version is running at the moment. A failing request, an error report in your monitoring tool, or a broken webhook, all of those interactions your services handle at each point in time run on one commit. So when something breaks, it is crucial to know which version is actually running.

Thankfully, for most types of services, this can be handled incredibly easily. I'll explore how we can annotate our deployments with the underlying commit SHA so we can add internal references to it, and make our investigations more focused.

With this improvement in place, you'll always know which version a failing request was running on, and where to look for potential fixes. You'll also immediately know if your bug fix is already live and the issue is gone, or still sticking around.

Adding the commit SHA to your containers

If you're building your container images using any OCI-compliant tool, you can supply build arguments, which are similar to environment variables, except they only stick around for the build phase, and only in the current build stage. Let's make it a bit simpler to understand:

You can define a build argument in your Dockerfile like this

FROM busybox
ARG user=somedefault
USER $user

FROM busybox AS next
...

As you can see, we define an argument, user, with a default value (somedefault). In the next step, we can use the variable. It is important to grasp that if we continue our build and get to the next build stage, we can no longer access the variable. We'd have to redefine it.

If the build completes, and we create a container running it, the variable is also gone. It only exists during build time. So we need a way to make it stay even when we run our service since we want to access it as an environment variable.

So in your final build stage, add the following

ARG commit_sha=unknown
ENV BUILD_COMMIT_SHA=$commit_sha

This will define a new build argument commit_sha, set its default value to unknown and pass it to an environment variable BUILD_COMMIT_SHA. Go ahead and build your container image with

docker build --build-arg commit_sha=$(git rev-parse HEAD) ...

or change your existing tooling to add the build argument.

Once this is done, you're all set. In your service, simply load the environment variable BUILD_COMMIT_SHA and use annotate all reports, requests, and other information you might have to investigate. If the commit SHA is too sensitive for you, pick a value you prefer, whatever works for you, as long as it helps you identify which version is running right now.

Next.js

With environment variables supported out of the box, simply add a public variable to your defaults in .env.local (or your respective defaults file)

NEXT_PUBLIC_BUILD_COMMIT_SHA=unknown

and make sure to declare the current commit SHA in your environment when building

NEXT_PUBLIC_BUILD_COMMIT_SHA=$(git rev-parse HEAD) next build

This should get you going 👍


I hope this short guide convinced you to add another quality of life improvement to your project of choice! If you have any feedback, questions, or suggestions, don't hesitate to reach out on Twitter or send a mail.