Designing and Scaling Distributed Systems

Building larger systems often requires connecting decoupled services, deploying containerized applications, and managing data across multiple distributed and replicated data stores. Deploying and managing these systems creates new challenges in terms of scalability, observability, and maintainability. This collection of articles covers various aspects of designing and scaling distributed systems, from communicating between services to handling long-running jobs and improving cache hit ratios.

Sharding high-throughput Redis without downtime

Every day, more and more software teams rely on Inngest for their core infrastructure which results in increased load on our core systems. When building distributed systems, vertical scaling can only get you so far. Because of that, we took on a project to horizontally scale by sharding our most crucial database, which serves more than 50,000 requests per second at low latency. As Inngest is critical infrastructure for many customers, we aim to avoid maintenance windows at all costs. In this post we’ll walk through how we sharded our function state database with zero downtime....Inngest, Systems Architecture, Redis, Sharding

Enhancing Scalability and Reducing Latency Without Missing a Beat

At Inngest, we serve hundreds of millions of events per day. Each event can trigger a function run comprising multiple steps. Users can return blobs and other data from these steps, necessitating incredibly fast reads and writes to our data store to keep latency as low as possible. Function runs can be delayed up to many months, so this data needs to be persisted over an arbitrary amount of time....Systems Architecture, Golang

Harnessing the Power of Go Interfaces for Decoupling and Scaling at Inngest

Interfaces hide implementation details by specifying the capabilities or behavior of an object. This concept has appeared in different programming languages over time, using different terms like \*traits\* or protocols....Systems Architecture, Golang

Zero-Downtime Migrations in Producer/Consumer Systems

Imagine you’re building an app and want to ensure that requests don’t feel slow for the end user. Let’s say you’re working on a signup flow that needs to send an activation email and update external systems like billing and a CRM. You don’t really need to block the request until all these steps have run to completion. Instead, you could decouple this workflow into a producer and consumer. When receiving the signup request, your API would simply enqueue a task to some distributed queue, which a worker service could then pick up and process....Systems Architecture

Derisking Product Rollouts with Feature Flags

In most successful companies, products and their underlying infrastructure are constantly evolving to changing needs and growing demand....Systems Architecture, Engineering Management, Product Development

Using Static Code Analysis to Share Types Between Services

Around four months ago, I shared a story of how we were trying to solve the problem of sharing code for Anzu. As we are building both our frontend and backend with TypeScript, we thought it should be easy enough to share parts between both sides, from types to functions. This road has been far bumpier than we expected, and today I’ll go into more detail about how we solved the issue in the end....TypeScript, Tooling, Anzu

Offloading Execution Flow to the Database

This post is the second part of a series on exploring the benefits and drawbacks of engineering for the application and database layer. If you haven’t done so, check out the first post on reducing database roundtrips....Systems Architecture

Balancing Normalization and Revision History

From early on, we’ve learned to deduplicate data and use concepts like foreign keys to make the most of our relational databases. Cascading operations are great to handle implicit updates and retain relational integrity, but somewhere, there is a case for denormalization....Systems Architecture

Handling Long-Running Jobs with AWS SQS

Decoupling services usually involves some form of communication, preferably asynchronous through queues that are populated by one service and consumed by another. AWS SQS offers regular and FIFO queues that help you with exactly that use case, but, as always, the devil is in the details....Systems Architecture, Amazon Web Services (AWS)

Reducing Database Round-Trips with Boring Queries

Note: This post is the first part of a series on exploring the benefits and drawbacks of engineering for the application and database layer....Systems Architecture

Communicating between services with gRPC

Setting up the communication layer between services can be a cumbersome task, even if they’re written in the same language, built on top of the same framework, and running on the same infrastructure. Every minute spent debugging weird connection issues, malformed payloads, and other parts that occasionally break could have been spent on building the actual product....Systems Architecture, gRPC, Golang

Paginating Large, Ordered Data Sets with Cursor-Based Pagination

In my guide on designing scalable APIs, I emphasized implementing strict limits to how many entries are sent back for endpoints that return multiple items. The idea was that as long as you’re not completely sure that the data to be returned is limited to a small number of items, always use pagination....Systems Architecture, GraphQL, PostgreSQL

Choosing an Implementation for Unique Identifiers

Picking the right underlying system for generating unique identifiers (read ID generator implementation) throughout your application is an important task early on. Even if it doesn’t seem that important, having to revert a bad decision made early can require a lot of time and effort you’d gladly spend on a thousand other things....Systems Architecture

Implementing Transactions with Side Effects

Transactions are an amazing concept: You perform some work, and only if everything succeeds, the results get persisted. If something in between fails, the system will simply roll back and thus undo all changes. Until a transaction is committed, no one can see its changes....Systems Architecture

Communicating between Java and Go using gRPC

Sometimes you're working between two systems, running in different processes, based on different programming languages. If you want to get instructions or data across those two, you'll need a layer that handles communication....Golang, Java, Systems Architecture

Efficient Caching with Reduced Roundtrips

Imagine you're building a service that heavily depends on content served by an external provider. Sending a request each time a resource is accessed would quickly lead to rate-limiting, so to provide an improved experience, we want to introduce caching....Caching, DataLoader, Systems Architecture

Boring Technologies, Reliable Foundations

In the ever-changing world of software engineering, keeping track of, not to mention specializing in, a broader range of areas and technologies has become increasingly difficult in the past years....Systems Architecture

Improving Cache Hit Ratios with Connected DataLoaders

The DataLoader concept has become popular with GraphQL, combining batching and caching of database requests, API calls, and any other resource fetching implementation....GraphQL

Efficient Caching with Reduced Roundtrips

Imagine you're building a service that heavily depends on content served by an external provider. Sending a request each time a resource is accessed would quickly lead to rate-limiting, so to provide an improved experience, we want to introduce caching....Caching, DataLoader, Systems Architecture

Versioning Features for Sustainable Product Evolution

When you're working on a product that is constantly evolving, adding, deprecating, and deleting features not only has to be communicated with your customers in advance, but often you have to maintain multiple versions....Evergreen, Systems Architecture, Product Development

Designing APIs to Last

In the era of Software-as-a-Service products, we have seen a transition from interacting through user interfaces to interacting through APIs. Some products are available mainly through such a communication layer, enabling automation, and seamless integration between services....Evergreen, Systems Architecture, GraphQL