On Friday 6 April at 16:04 BST, we experienced a service outage. For a period of 27 minutes, the GoCardless API and Dashboard were unavailable, and users were unable to set up a Direct Debit via our payment pages or connect their account to a partner integration through our OAuth flow.
Submissions to the banks to collect payments and pay out collected funds were unaffected.
We’d like to apologise for any inconvenience caused to you and your customers. As a payments company, we know how important reliability is to our customers, and we take incidents like this extremely seriously. We’re completing a detailed review of the incident and taking the required steps to improve our technology and processes to ensure this doesn’t happen again.
All of our most critical data is stored in a PostgreSQL database. When we make changes to certain tables in that database (i.e. create or update a row), we use a trigger to keep a separate record of exactly what changed. We use this “update log” to enable data analysis tasks like fraud detection.
Each entry in the log has an automatically-generated sequential ID (1, 2, 3, and so on). This ID is stored using the
serial type in the database, which means it can be a value between 1 and 2147483648.
At 16:04 on Friday 6 April, we hit this upper limit, meaning we could no longer write to the “update log” table. In PostgreSQL, when a trigger fails, the original database write that triggered it fails too. This caused requests to our application to fail, returning a
500 Internal Server Error.
This issue also affected API requests (including those from the Dashboard) which only appear to read data (e.g. listing your customers or fetching a specific payment), since authenticated requests update access tokens to record when you last accessed the API.
Having identified the root cause of the problem, we disabled the trigger which sends writes to the “update log”, thereby restoring service.
We’ve resolved this problem for the future by storing the IDs for our “update log” using the
bigserial type, which allows values up to 9223372036854775807. This is effectively unlimited, and can be expected to provide enough IDs to last millions of years.
In the next few days, we’ll be running a full post-mortem to better understand:
We’ll publish the results of this post-mortem in a follow-up post within the next 4 weeks.
In this post, I’ll talk about how we changed the way we work over the last 9 months to build truly global software and introduce a localisation process which allows us to move quickly and deliver real value for customers.
We wanted to provide a great experience for our users, whatever language they speak — but it was imperative to do so in a way that didn’t slow us down as we continue to build out our product. When we get processes like this wrong, we not only make our team’s work harder than it needs to be, but we place a drag on what we care about most: delivering value for our users.
There’s a whole other post I could write about the intricacies of that process and how we’ve invested in sourcing skilled translators and ensuring we have perfect translations with quality assurance (QA) processes - but in this post, we’ll focus on the developer workflow.
Business services is fast-paced, dynamic and competitive. But being busy isn’t enough. Whatever field your agency operates in, evidence of your success lies in your company’s bank balance, as much as in the volume of work you’re tackling. Having lots of projects on the go and more in the pipeline is well and good. But if you’re not getting paid on time your business isn’t truly healthy - or ready to grow and exploit new opportunities.
The nature of the work many agencies undertake can exacerbate issues with cash flow. Projects often take some time to complete and can prove elastic, with clients requesting unexpected last minute changes. Plus, larger client firms might insist on long payment terms, leaving an agency waiting for money for months after work has been delivered.
It doesn’t have to be this way. There are several paths to improving your business’s liquidity. What exactly should an ambitious agency owner be doing to ensure they have a healthy cash flow? And how can they use that strengthened position to scale their company up?
Over the past few weeks, I’ve been speaking with energy supply businesses who are curious about open banking and its implications for the sector. At GoCardless, we're using open banking as a platform to improve payments for our customers. In this blog, we look at how open banking could impact energy suppliers and consumers.
Open banking is a new set of software standards for banks set by the Open Banking Implementation Entity (OBIE), an entity created by the UK’s Competition & Markets Authority (CMA) in 2016. The CMA’s goal is to drive competition and innovation in UK retail banking.
The new standards require UK banks to create APIs that allow two new types of services to be run by authorised, FCA regulated third parties (like GoCardless). These third parties will fall into two categories:
Account Information Services Providers (AISP): AISPs will be able to access consolidated information about payment accounts held by a user. Once authorised by the customer, AISPs will be able to access information such as current balance and transaction history.
Payment Initiation Services Providers (PISP): PISPs will be able to access a user's online payment account (with their consent) to initiate the transfer of funds on a user's behalf in near real-time. This represents an alternative way to pay online, avoiding the need for credit card or debit cards. These services are not yet widely used in the UK, but are commonly used in Europe.
But what does this mean for the energy market?
The culture of late payments has become so ingrained in the UK that spending time, money and resource chasing payments has, for many, become business as usual.
Take a look at the graphic below to see, in numbers, the state of late payments in the UK, how this compares to other European countries and if with new legislation, there is any sign of things improving.
We’re reaching a tipping point for bank-to-bank payments and PSD2 might just push us over the edge.
Direct Debit, the most common means of collecting bank-to-bank payments, was devised in the 1964 by a Unilever executive, as an automated way to collect recurring, variable payments from ice cream vendors, without having to ask permission each time.
There are several factors that have contributed to the growth of Direct Debit in the UK and Europe:
Third-party providers like GoCardless have opened up access to Direct Debit to thousands of SMEs in the UK who could not previously meet the revenue and bond criteria set out by banks. These providers act as a merchant account for businesses, developing and managing banking relationships on their behalf.
Ease of use
More commercial providers offering Direct Debit has led to significant improvements in user experience. While the former paper-based Direct Debit system was clunky and disconnected from the rest of a business’ workflow, GoCardless now gives merchants a simple, automated way to collect payments, through an app within their billing or CRM software, through an online dashboard or by building their own integration with our REST API.
The growth of the ‘subscription economy’ in the last decade has led businesses to seek payment solutions more suited to a recurring revenue business model. Bank-to-bank mechanisms like Direct Debit allow these businesses to collect recurring payments against a subscription plan with a single mandate, while reducing involuntary churn and transaction costs (payment failure rates and transaction costs are lower for Direct Debit than for cards).
The problem of gender diversity in the tech industry is not new and there are many businesses and charities doing great work to make the industry more inclusive. For example, Code First Girls' 2020 campaign has already trained 5000+ women how to code for free and the newly-launched Fintech Parity Pledge encourages those working in events to only speak at, attend, and organise events that promote diversity.
Despite the success of these campaigns, there is still a long way to go, with some organisations predicting that the gender gap will worsen if more is not done. According to Code First Girls, female programmers and software developers made up only 3.9% of tech and telco professionals in the UK in 2017, down from 10% in 2007.
To celebrate International Women’s Day, we sat down with some of the women in GoCardless’ Product Development team to hear about their experiences as women in tech (and fintech, a sector in which women represent only 29% of the workforce), and to get their thoughts on what can be done to encourage more women into the industry.
Accountancy firms might spend their days advising clients on their finances, but even they are aren't immune from the problem of late payments. The average professional service firm is owed £54k in outstanding payments, according to new research by data specialists Dun & Bradstreet.
Having a challenging conversation with your client about money isn't most people's idea of fun, but most small firm leaders will need to do it at some point. With that min mind. we've compiled some tips to help you chase your client’s unpaid invoice (without chasing your client away).
When chasing an unpaid invoice, it’s important to remember that payment is already late. So, don’t procrastinate. By making contact with your client as soon as the invoice becomes overdue, your firm may be able to close the payment gap by presenting more flexible payment options, such as changing the payment date or offering instalment payments through Direct Debit. Whilst alternative payment routes don’t guarantee payment, they do move the dial in the right direction.
Often, a client who is trying to delay payment will wait until payment is due before advising you that they didn’t receive your invoice.
Your firm has delivered a service, your client hasn’t paid for it and all you can hear are crickets. So, it’s easy to think that non-payment is intentional, but it may not be. Genuine oversights happen all the time, so give your client the benefit of the doubt (especially first-time offenders).
Once the invoice payment deadline has passed, a junior staff member from your Accounts department should give your client a gentle nudge by sending a firm – but friendly – email with a copy of the invoice attached. The email should say that payment is overdue and your client must arrange payment as soon as possible or contact your firm if they’re in financial difficulty. More often than not, a subtle reminder is all that’s needed to spur an otherwise motionless client into action.
March is set to be a busy month here at GoCardless as members of the team head out to various events across the UK and beyond.
With events season really starting to kick-off, take a look below to see where you can find the GoCardless team in March.