Building APIs: lessons learned the hard way

This post is about the lessons we’ve learned building and maintaining APIs over the last three years. It starts off with some high-level thoughts, and then dives right into the detail of how and why we've made the design decisions we have, while building our second API - GoCardless Pro.

The problem with APIs

The hard thing about APIs is change. Redesign your website and your users adapt; change how timestamps are encoded in your API and all of your customers' integrations break. As a payments API provider even a single broken integration can leave thousands of people unable to pay.

As a fast-moving startup we're particularly poorly positioned to get things right first time. We're wired to constantly iterate our products: ship early then tweak, adjust, and improve every day. As a startup the uncertainty goes even deeper: we're always learning more about our customers and making course corrections to our business.

Building dependable APIs in this environment is hard.

Different types of change

The most important lesson we've learned is to think about structural changes differently to functionality changes.

Structural changes affect the way the API works, rather than what it does. They include changes to the URL structure, pagination, errors, payload encoding, and levels of abstraction offered. They are the worst kind of changes to have to make because they are typically difficult to introduce gracefully and add little to existing customers. Fortunately they're also the decisions you have the most chance of getting right first time - an API's structure isn't tied to constantly-evolving business needs. It just takes time and effort, and is discussed in "Getting your structure right" below.

Functionality changes affect what the API does. They include adding new endpoints and attributes or changing the behaviour of existing ones, and they're necessary as a business changes. Fortunately they can almost always be introduced incrementally, without breaking backwards compatibility.

Getting your structure right

In our first API we made some structural mistakes. None of them are serious issues, but they have led to an API that is difficult to extend due to its quirks. For instance, the pagination scheme relies on limits and offsets, which causes performance issues. We also had frequent discussions about whether resources should be nested in URLs, which resulted in inconsistencies.

Before we started work on GoCardless Pro, we spent a lot of time laying the structural foundations. Thinking about the structure of GoCardless Pro was helpful for several reasons:

• we were forced to think upfront about issues that would affect us down-the-line, such as versioning, rate-limiting, and pagination;

• when implementing the API, we could focus on its functionality, rather than debating the virtues of PUT vs PATCH;

• consistency across the API came for free, as we’re not making ad-hoc structural decisions.

We adopted JSON API as the basis for our framework and put together a document detailing our HTTP API design principles. The decisions made came from three years of experience running and maintaining a payments API, as well as from examining similar efforts by other companies.

Amongst other things, our framework includes:

• Versioning. API versions should be represented as dates, and submitted as headers. This promotes incremental improvement to the API and discourages rewrites. As versions are only present in incoming requests, WebHooks should not contain serialised resources - instead, provide an id to the resource that changed and let the client request the appropriate version.

• Pagination. Pagination is enabled for all endpoints that may respond with multiple records. Cursors are used rather than the typical limit/offset approach to prevent missing or duplicated records when viewing growing collections, and to avoid the database performance penalties associated with large offsets.

• URL structure. Never nest resources in URLs - it enforces relationships that could change, and makes clients harder to write. Use filtering instead (e.g. /payments?subscription=xyz rather than /subscriptions/xyz/payments)

This list is by no means exhaustive - for more examples, check out our full HTTP API design document on GitHub.

When starting work on a new API we encourage you to build your own document rather than just using ours (though feel free to use ours as a template). The exercise of writing it is extremely helpful in itself, and many of these decisions are a question of preference rather than "right or wrong".

Functionality changes

You can’t ever break integrations, but you have to keep improving your API. No product is right first time - if you can’t apply what you learn after your API is launched you’ll stagnate.

Many functionality changes can be made without breaking backwards compatibility, but occasionally breaking changes are necessary.

To keep improving our API whilst supporting existing integrations we:

• Use betas extensively to get early feedback from developers who are comfortable with changes to our API. All new endpoints on the GoCardless Pro API go through a public beta.

• Version all breaking changes and continue to support historic versions. By only ever introducing backwards-incompatible changes behind a new, we avoid breaking existing integrations. As we keep track of the API version that each customer uses, we can explain exactly what changes they'll need to make to take advantage of improvements we've made.

• Release the minimum API possible. Where we have a choice between taking an API decision and waiting, we choose to wait. This is an unusual mentality in a startup, but when building an API, defaulting to inaction is probably the right approach. As we’re constantly learning, decisions made later are decisions made better.

• Introduce “change management” to slow things down. This is not our typical approach - “change management” is a term that makes many of us shudder! But changes to public APIs need be introduced carefully, so however uneasy it makes us, putting speed bumps in place can be a good idea. At GoCardless, all public API changes need to be agreed on by at least three senior engineers.

Stop thinking like a startup

To sum up: when building an API, you need to throw most advice about building products at startups out the window.

APIs are inflexible so the advice founded on change being easy doesn’t apply. You still need to constantly improve your product, but it pays to do work up-front, to make changes slowly and cautiously, and to get as much as possible right from the start.