Debugging Google Cloud Pub/Sub services in production

If you have ever experienced operational incidents like services seemingly not receiving or handling Pub/Sub messages correctly in production and you're uncertain where the problem originates from, it would be really helpful to observe the stream of messages flowing through the system in real-time. But sometimes, you don't necessarily have the required time or the possibilities to deploy another service version tracking incoming messages in the receiver for you to watch changes. This post focuses on handling exactly that problem and offers some solutions to monitoring messages using Google Cloud's Pub/Sub product and related tools.

How Google's Pub/Sub works

To outline it roughly, Google's Pub/Sub architecture primarily consists of two parts: topics and subscriptions. While you, the publisher, can simply push data to specific topics, all messages to said topics are forwarded to the respective subscriptions, where they are consumed or re-sent if not acknowledged (until expired). There are two types of subscriptions you can choose from: pull-based subscriptions are, as the name suggests, simply pulled by services in an event-based model, while push-based subscriptions post the message data to a remote address. If you are only using pull-based subscriptions, you can skip the following section and continue right to the next part.

When receiving messages using push-based subscriptions, the following workaround is especially important for our debugging situation: Because it is not possible to hook into push subscriptions, we have to create another temporary subscription (pull), to be able to receive published messages without having to log the message once it's delivered to our service. By doing this, we basically "duplicate" the message to be piped into a new subscription where it is pulled from, read and acknowledged.

Creating a new (temporary) subscription

You can either use the Google Cloud Console (web dashboard) or the gcloud CLI to quickly create a new subscription for your topic.

Using the Google Cloud Console

Using the gcloud CLI

$ gcloud beta pubsub subscriptions create readonly-subscription --topic sample-topic
Created subscription [projects/pubsub-post-sample-project/subscriptions/readonly-subscription].

Reading incoming Pub/Sub messages

Luckily, we can utilize the gcloud CLI to pull, read and acknowledge newly published messages and print them to our terminal. To do that, head over to your terminal and run

gcloud beta pubsub subscriptions pull --wait --limit 10 readonly-subscription
┌─────────────┬─────────────────┬───────────────┬─────────────────┐
│     DATA    │    MESSAGE_ID   │   ATTRIBUTES  │     ACK_ID      │
├─────────────┼─────────────────┼───────────────┼─────────────────┤
│ hello-world │ 512791743025732 │ source=manual │ <ACK_ID>        │
└─────────────┴─────────────────┴───────────────┴─────────────────┘

To get further details on running the pull command on subscriptions, check gcloud beta pubsub subscriptions pull --help.

And that's already it. With this short guide, you are now able to fetch the latest messages once they are published 👍 If you have got any questions, suggestions or other feedback, don't hesitate to send a mail.