When working on projects in the past, I’ve always wondered about the perfect time to lock down interfaces. In the first steps of building an idea, you’re not quite sure what data you need to exchange yet and which operations should be exposed via APIs. Perhaps you don’t even know which transport protocol you want to use (e.g. whether you want a REST API or GraphQL).
Since we departed from building applications with a single PHP file (well, not all of us, all the power to the people who are still winning with that stack), we’ve added an increasing number of indirections to our stack: Data flows from databases to backend services to frontend applications.
In the backend, you transform your database layout to an intermediate data structure that gets then transformed for transport. On the frontend side of things, you receive data in one format but have to transform it into a representation that fits your state management solution.
With this simple layout, you already have to deal with four steps of data transformation, just to get data from your database to be displayed to a user. Sure, using libraries like Prisma makes data access type-safe, and GraphQL gets you a type-safe client and server structure, but you still spend an awful amount of time converting data.
Well, indirection isn’t all bad. In fact, when scaling your architecture, you’ll keep hearing one phrase: decoupling. You’re told to decouple your services by adopting parts of event-driven systems, so that you can scale more easily and have teams working on different parts of the system without depending on each other.
When you’re at a point where stability becomes important for your business, adding layers through decoupling makes a difference: Stable interfaces provide guarantees, but you pay the price in velocity.
As an active reader of this blog, you might think “oh we’re back to velocity, what an unexpected twist”.
When adding a column to the database requires updating dozens of files throughout different services, you really should be asking yourself if you want this level of rigidity. Chances are that you’re still early enough that velocity should be the main focus and trading off some stability is completely fine, as you don’t have customers that depend on stable APIs yet.
Of course, the goal should not be to reject stability at all costs, there’s a gradual transition to stability once you reach a certain size.
Let’s go over different stages of this transition, from the early days where you need to move as fast as possible, to later stages where interface stability is key, comparing both benefits and drawbacks of each workflow.
Stage 0: Database to Frontend
In the beginning, you might as well transfer raw database objects to the client. Sure, this is neither great, terribly secure, nor future-proof, but it does the job and there’s no thinner interface than no interface at all. Fetch your data from the database, and reuse the type you parse everything into. Bonus points if you use a library to offload the database interaction and parsing work, and maybe even use type generation to take care of type safety.
When set up, you have a playground ready to use. Adding data to your site means writing a query and returning the data straight to your client, as our predecessors envisioned.
If you’re still thinking this is way too half-baked, you’re worrying about the wrong stability. Validate your idea first, then make sure it scales. If you’re working in a regulated area, skip ahead to stage 3.
Adding new features involves a few lines of code and service restarts, everything is pretty easy to grasp.
Stage 1: Versioned API
After some time your first customers will have started using your product. If you’re building something for developers, you may probably want to expose your API. In that case, rewriting the docs every time you do a breaking change probably isn’t a great experience for anyone, so it’s time to make sure you have a policy for breaking changes.
Ideally, you add some versioning format, so customers can keep using an old version until they’re ready to move (don’t be afraid to nudge (or force) them to adopt the newest version once in a while, you don’t want to end up maintaining multiple versions of your system longer than necessary). Prefer avoiding breaking changes if possible, use a feature preview to test while staying flexible enough to make changes.
Adding new features now involves checking if the change breaks the API for existing users, in which case a new version should be released, and otherwise adding the related fields to the API.
Stage 2: Frontend Indirection
Our frontend is growing in complexity and the developers think it makes sense to upgrade the state management. For this, they might need some custom types exclusive to the frontend, including parsing and validation. Since your APIs are safe to be consumed and no breaking changes are in sight, everyone feels safe. If you do end up having to make a breaking change, make sure the frontend handles versioning correctly. Complexity will inevitably rise.
Adding new features now involves updating the frontend types and state management accordingly.
Stage 3: Backend Indirection
With our frontend becoming more advanced, we need another layer of indirection for the backend. Instead of data flowing to the database and back, we may become interested in event processing, contextual validation, and output formats like webhooks and audit logs.
Adding new features now involves updating event handlers and their related types, making sure the validation logic captures our changes and keeping third-party systems updated.
Each layer of indirections gets us closer to a stable system with clear procedures laying out how changes should be made at the cost of involving more people’s time and context to keep in your mind. These layers act as guards to keep everyone accountable and prevent breaking the experience of people who are paying you for a stable product. On the flip side, if there are no strong reasons, you have to jump through more hoops than necessary.
When considering adding indirection, estimate the added difficulty (lost velocity) in making changes to the system in your cost-benefit calculation.