As a payments company, our APIs need to have as close to 100% availability as
possible. We therefore need to ensure we’re ready for whatever comes our way:
from losing a server without bringing the API down, to knowing how to react if a
company laptop is compromised.
To accomplish this we run GameDay exercises. What you will read below is our
version of a GameDay. We hope that by sharing how we do GameDays we can give you
a starting point for running your first GameDay.
The UK tech scene is becoming increasingly competitive, with companies constantly on the hunt for great developers to join their ranks. With so much choice out there, developers can afford to be ultra-picky when choosing a job.
Today’s tech companies offer a wide range of perks to attract the best developers ahead of the competition. These can range from office treats such as well-stocked beer fridges and team lunches, to joining bonuses, conference budgets, and company holidays.
It’s great having office pizza for lunch. But when considering that all-important next career move, many developers also search for more meaningful factors in their work environments.
GoCardless recently won the Techies 2017 prize for ‘Best Place for Developers to Work’. In the process of creating a great work environment for all our people, we’ve learned a few things about what developers want in a role.
We chatted to some of our engineering team to find out what they look for in a role and what attracted them to GoCardless.
Our API is at the heart of everything we do. Not only does it allow developers to build powerful Direct Debit solutions, but it also powers the GoCardless Dashboard along with our integrations with partners like Xero, Teamup and Zuora.
As engineers, we know firsthand the importance of having great resources at your disposal when you’re getting started with an API. So back in September we kicked off a major project to revamp our developer onboarding experience. We wanted to build something we could be really proud of and which would also delight our customers.
In this blog post, the first of a series of two, we’ll take you through the journey of building our new developer site from idea to delivery.
Interviewing is hard. Both the company and the candidate have to make an incredibly important decision based on just a few hours’ worth of data, so it’s worth investing the time to make those precious hours as valuable as possible.
We recently made some changes to our DevOps interview process, with the aim of making it fairer, better aligned with the role requirements, and more representative of real work.
We started by defining the basics of our DevOps roles. What makes someone successful in this role and team? What are the skills and experience that we're looking for at different levels of the role?
It was important that the process would work for candidates with varying experience levels, and so it needed to be flexible and clear to assess skills at each of these levels.
The skills we’re looking for fall into three broad categories: existing technical knowledge (e.g., programming languages), competency-based skills (e.g., problem solving), and personal characteristics (e.g., passion for the role, teamwork and communication skills). After defining these skills, we mapped out how we would assess them at each stage of the interview process.
I’m Tim, and I’m a software engineer at GoCardless. I’ve been here for about four and a half years. I work on our UX team, building customer-facing bits of GoCardless, such as our dashboard and developer API. My team focuses on making them as powerful and easy to use as possible.
The early days
I first joined the team back in 2012, just a few months after GoCardless had launched into beta. I was attracted by the boldness of what the company was trying to do: making life better for small businesses and disrupting banks’ traditional monopoly. Since then I’ve worked in a variety of roles across the company, from setting up and running our customer support operation to running our partnerships team, to where I am now.
As developers, we work on features that our users interact with every day. When you're working on the infrastructure that underpins those features, success is silent to the outside world, and failure looks like this:
Recently, GoCardless moved to a container-based infrastructure. We were lucky, and did so silently. We think that our experiences, and the choices we made along the way, are worth sharing with the wider community. Today, we're going to talk about:
deploying software reliably
why you might want a container-based infrastructure
what it takes to reliably run containers in production
We'll wrap up with a little chat about the container ecosystem as it is today, and where it might go over the next year or two.
The GoCardless API allows you to manage Direct Debit payments via your own website or software. When a customer signs up for your services they can give you a Direct Debit authorisation online. Your integration can then create and manage payments and subscriptions automatically - there’s no need to manually add a new customer to GoCardless. Our API provides you with full flexibility when it comes to payment creation, and we offer it to all of our merchants at no extra cost.
In this blog post we’ll guide you through the steps needed to use our API, from customer creation to taking your first payment.
Let’s look at how Direct Debit payments work and how the GoCardless API is organised. In order to charge a customer’s bank account, you will first need their authorisation to collect payments via Direct Debit. This can be via our secure online payment pages or, if you’re using GoCardless Pro, you can take complete control of the payment process by hosting the payment pages on your own website.
On Thursday, we experienced an outage of 1 hour and 40 minutes that affected all our services.
During that time you may have seen brief service recovery but for the most part our service was unavailable.
We're pleased to announce the release of ActiveRecord::SaferMigrations, a library to make changing the schema of Postgres databases safer. Interested how? Read on.
Previously, we looked at how seemingly safe schema changes in Postgres can take your site down. We ended that article with some advice, and today we want to make that advice a little easier to follow.
In a nutshell, there are some operations in Postgres that take exclusive locks on tables, causing other queries involving those tables to block until the exclusive lock is released1. Typically, this sort of operation is run infrequently as part of a deployment which changes the schema of your database.
For the most part, these operations are fine as long as they execute quickly. As we explored in our last post, there's a caveat to that - if the operation has to wait to acquire its exclusive lock, all queries which arrive after it will queue up behind it.
You typically don't want to block the queries from your app for more than a few hundred milliseconds, maybe a second or two at a push2. Achieving that means reading up on locking in Postgres, and being very careful with those schema-altering queries. Make a mistake and, as we found out, your next deployment stops your app from responding to requests.
After making some changes to our Postgres setup, we started noticing occasional errors coming from deep within ActiveRecord (Rails’ ORM). This post details the process we went through to determine the cause of the issue, and what we did to fix it.
First, it’s important to understand the changes we made to our Postgres setup. Postgres connections are relatively slow to establish (particularly when using SSL), and on a properly-tuned server they use a significant amount of memory. The amount of memory used limits the number of connections you can feasibly have open at once on a single server, and the slow establishment encourages clients to maintain long-lived connections. Due to these constraints, we recently hit the limit of connections our server could handle, preventing us from spinning up more application servers. To get around this problem, the commonadvice is to use connection pooling software such as PgBouncer to share a small number of Postgres connections between a larger number of client (application) connections.
When we first deployed PgBouncer, we were running it in “session pooling” mode, which assigns a dedicated Postgres server connection to each connected client. However, with this setup, if you have a large number of idle clients connected to PgBouncer you’ll have to maintain an equal number of (expensive) idle connections on your Postgres server. To combat this, there is an alternative mode: “transaction pooling”, which only uses a Postgres server connection for the duration of each transaction. The downside of transaction pooling is that you can’t use any session-level features (e.g. prepared statements, session-level advisory locks). After combing through our apps to remove all usages of session-level features, we enabled transaction pooling.
Shortly after making the switch, we started seeing (relatively infrequent) exceptions coming from deep within ActiveRecord: NoMethodError: undefined method 'fields' for nil:NilClass. We also noticed that instances of this exception appeared to be correlated with INSERT queries that violated unique constraints.
GoCardless Ltd.338-346 Goswell Road,
GoCardless (company registration number 07495895) is authorised by the Financial Conduct Authority under the Payment Services Regulations 2009, registration number 597190, for the provision of payment services.