Interfaces hide implementation details by specifying the capabilities or behavior of an object. This concept has appeared in different programming languages over time, using different terms like *traits* or protocols.
The same rules apply in Go, take the classic example of io.Reader:
type Reader interface {
Read(p []byte) (n int, err error)
}
This interface is used everywhere. It doesn’t matter if you’re reading from a file, a network socket, or a data structure in the same process: As long as you need to read bytes from a source, the interface can be used.
This is amazing because it allows decoupling the implementation from the downstream consumers. Your code shouldn’t care about how exactly a file is read from disk, or whether it is read from disk at all.
Interfaces enable many use cases, and in Go, they’re very idiomatic to use whenever it makes sense. One rather interesting case I’ve run into recently, was decoupling data access in our production codebase at Inngest.
An intro to Inngest
Inngest orchestrates durable functions and workflows, hosted on any service that can accept HTTP requests. Each function includes one or more steps triggered by events.
During a function run, we need to store details like the return value of your steps so we can load them on the next step. Here’s an example function that showcases Inngest’s capabilities: When a user signs up, Inngest runs steps to load the user data, send a welcome email, and wait for a post creation event, triggering further actions based on the outcome.
export default inngest.createFunction(
{ id: "activation-email" },
{ event: "app/user.created" },
async ({ event, step }) => {
const user = await step.run("load-user", async () => {
return await db.loadUser({ email: event.user.email });
});
await step.run("send-welcome-email", async () => {
return await sendWelcomeEmail({ email: user.email, name: user.name });
});
// Wait for an "app/post.created" event
const postCreated = await step.waitForEvent("wait-for-post-creation", {
event: "app/post.created",
match: "data.user.id", // the field "data.user.id" must match
timeout: "24h", // wait at most 24 hours
});
if (!postCreated) {
// If no post was created, send a reminder email
await step.run("send-reminder-email", async () => {
return await sendReminderEmail({
email: event.user.email,
name: user.name,
});
});
}
}
);
Storing Run State
To handle long-running functions, Inngest stores state details like loaded user data. With waitForEvent potentially delaying execution for extended periods, our system must manage state efficiently.
Previously, our execution logic was tightly coupled to Redis for storing function run state. As Inngest Cloud grew, we needed scalable solutions like sharding and caching. We unified state access behind a state.RunService interface, ensuring flexibility and scalability.
Consolidating Data Store Access
By creating new interfaces for state management, we created strong guarantees about the expected functionality that each implementation needed to follow. After merging this change, we started with the real work to prepare our system for the next stage.
To gracefully handle increasing system load, we needed to address two primary concerns:
- Too many connections to the backing data store
- Running out of storage capacity
High connection counts can strain systems like our highly-available data store used to persist function run state in our Cloud environment. This is a known problem. For example, Postgres creates a new backend process every time a connection is requested.
The solution here is to use a connection pooler like PgBouncer which acts as a proxy between your application and the Postgres instance. PgBouncer simply maintains a pool of opened connections, ready to be used for incoming client connections. Once all service connections are in use, new client connections need to wait until one server connection completes.
You might think “Why don’t you just use an application-level pooler?”, and that’s a great question! Instead of running one PgBouncer instance, you might initialize a connection pool in each application process, preventing it from establishing an unbounded number of connections. This works great as long as you limit the number of application processes. Once you create new services or auto-scale your production deployment, you might establish more connections than you’ve ever planned for. That’s why this approach is very limited in large, distributed systems.
To manage increasing connections to our data store and prevent storage issues, we introduced a coordinating service. This service, implementing the state interface, allowed us to handle connections efficiently and scale seamlessly.
Previously, replacing all state access logic with a new implementation would have taken weeks, not to mention the risk we would have taken in potentially missing one or two locations in our codebase.
With the new interface, we simply swapped the direct connection logic for the coordinator service in our production environment. Admittedly, we added more precautions for rolling out to all customers, but the idea was the same.
Interfaces saved the day.
Thanks a ton to Darwin, Jack, and Tony at Inngest for supporting me on this effort, it’s been super exciting to future-proof one of the most important pieces of our infrastructure.
If you’re looking for ways to build internal tools and workflows or want to implement multi-tenant queuing, please give Inngest a try!