Incident response: email reminders incorrectly sent to customers

Yesterday afternoon on the 4th July 2017, between 2pm and 2:35pm BST (British Summer Time), we sent a number of email reminders to end customers (those wanting to pay one of our merchants), asking them to authorise historic direct debit mandates. The emails were sent to individuals who received a request from one of our merchants to set up a mandate, but who never completed the original request.

All our systems remain secure and uncompromised, no third parties were involved, and we have put in place changes to fix the root cause of this issue. We apologise for any inconvenience caused.

What should I do?

What you should do depends on whether you’re an end customer who received an email from us on 4th July or a merchant collecting money through GoCardless.

For end customers

If you haven’t already taken action, please ignore the email you received. We have deactivated the link within it.

If you opened the link and completed the form, and you don’t want to set up payments to the merchant, please contact the merchant or your bank to cancel the Direct Debit mandate, or alternatively contact us if you have any queries.

For merchants

If your customers were impacted by the issue, we’ll be in touch with more detailed information.

We’ll be letting you know which customers were affected, and what options you have for handling any action taken by your customers in relation to the emails.

As always, our support team is happy to help.

In detail: how did this happen?

On the ‘Customers’ tab of our dashboard, we have a ‘Remind all’ button, which sends a reminder to all your customers who haven’t finished setting up a Direct Debit mandate with you.

As part of our phone support service, our staff can trigger these emails on behalf of a merchant.

The combination of a bug in our dashboard API and the way our dashboard works when staff are acting on behalf of merchants led to reminder emails being sent for all merchants that had pending customers, rather than just the merchant the support agent triggered reminders for.

As soon as we found out about the issue, we halted sending any further emails, triaged the bugs, and began working on our response.

We have fixed the bug, and performed multiple tests to ensure the fix is working as intended.

What’s next?

We want to reiterate our apology for the inconvenience caused by this issue.

As part of our incident response process, we’ll be conducting a detailed investigation into the issue and our response to it. We’ll be focused on finding any improvements we can - in both our systems and processes - that will reduce the likelihood and impact of this kind of issue happening again.

If you have any questions after reading this, please feel free to contact our support team.

New API Version - 2015-04-29

We last released a new version of the GoCardless Pro API back in November. Since then we’ve made countless improvements and recently we've been working on several new features which warrant the release of a new version.

Version 2015-04-29 is being released on Wednesday, along with a corresponding update to our dashboards. For the majority of integrations, the upgrade will be extremely simple (see upgrading).

The new API docs are available at: https://developer.gocardless.com/pro/2015-04-29/

We will continue our support for v2014-11-03 until the 1st of July, 2015.

Continue reading...

Need help upgrading or have any questions?
Get in touch
in Engineering

Visualising GoCardless' UK Growth

The above animation is a time lapse of customers using GoCardless for the first time. It covers the last 3 years and only maps the UK for now, with each red dot representing a new customer joining GoCardless, then becoming a blue dot for the remainder of the clip. It works even better if you view it full-screen.

Continue reading...

We’re hiring developers
See job listing

Tier One Design Mobile App

GoCardless Mobile App

Here at GoCardless we love developers, especially when they put together something cool with our API.

We've tried to make our API as flexible as possible to encourage new and interesting use cases. Previously that mostly meant web applications, but no longer!

The clever folks at Tier One Design have created an iPhone App which allows your to see a summary of all your GoCardless information on the go. You can keep up to date with all your customers, bills, subscriptions and account information all in one place right on your phone.

We love hearing from people about their experience developing with GoCardless. Have you created something awesome recently, or are you planning to integrate GoCardless with your app? Let us know!

Update (10th July 2013):

Tier One Design have just released a major update: you can now take payments from within the app. Check it out on the App Store.

Interested in building something with GoCardless?
Read the API docs
in Engineering

Cooking up an Office Dashboard Pi

Our Raspberry Pi Developer Dashboard

Since GoCardless started hiring a lot of new faces we've been looking for ways to keep everyone in touch with what's going on. One part of the solution has been adding dashboard screens around the office.

Putting together your own metrics dashboard is actually pretty simple and yields a lot of benefits. This post is a full how-to guide for building your own with a Raspberry Pi, an HDTV and a bunch of hackery.

Step One - Buy your ingredients

Raspberry Pi Components

TV Components

The TV supports a USB connection, so using the USB->Micro USB adapter, we can actually power the Pi without needing any additional wires going to the mains. Combine this with using a Wifi adapter instead of an ethernet cable and you can attach the Raspberry Pi to the back of the TV without any visible wires. Sweet.

For other configurations see here

Step Two - Prepare your filling

For Dashing Dashboards, use Dashing...

There were several options available to us for creating data dashboards but in the end, the one which seemed most flexible whilst remaining easy to implement was Dashing built by the awesome guys at Shopify.

Dashing allows you to very easily create jobs to pull and generate metrics which you can then send to pre-built (or custom) widgets on any number of customized dashboards in real time.

I won't go into more detail on how Dashing actually works, suffice to say it's awesome and you should check it out.

If you want to get hacking quickly, look at the Getting Started guide on the github page.

Persistance

The one thing we found was missing for us was persistence. For regularly updated metrics, vanilla dashing is great, if you reboot, you see new data in seconds. However for longer-interval updates (CI status, Deployments etc), when screens are turned off or Pis rebooted we ended up with blank dashboards in the interim.

The workaround we devised was very simple and involved setting Dashing's history to a Redis hash substitute instead of a standard ruby hash.

To do this you will need to add redis-objects to your Gemfile:

# Lets us persist data across reboots
gem 'redis-objects'

and then in config.ru add:

  # Redis URI is stored in the REDISTOGO_URL environment variable
  # Use Redis for our event history storage
  # This works because a 'HashKey' object from redis-objects allows
  # the index access hash[id] and set hash[id] = XYZ that dashing
  # applies to the history setting to store events
  redis_uri = URI.parse(ENV["REDISTOGO_URL"])
  Redis.current = Redis.new(:host => redis_uri.host,
                            :port => redis_uri.port,
                            :password => redis_uri.password)

  set :history, Redis::HashKey.new('dashing-hash')

Continuous Integration

We've recently started using CircleCI for our Continuous Integration and we really wanted a visualization of our CI status where everyone could see it instantly.

Dashing makes it super easy to add new widgets and since CircleCI has an API, it was relatively easy to come up with an integration of the two, resulting in these widgets:

Our Raspberry Pi Developer Dashboard

I've open sourced both our Single Panel and List Style widgets; feel free to customize them and add your own improvements!

Step Two - Bake your Pi

1. Image the SD card

I was doing all this on OSX so I followed the RPi Easy SD Card Setup guide and installed Raspbian Wheezy.

2. Get connected

The whole idea of this is to have the Raspberry Pi hidden behind the screen so trailing Ethernet cables isn't ideal. Luckily the Pi supports a range of Wifi adapters.

The Edimax wireless adapter I use eats a USB port but since we don't need it anyway that's not a problem. After plugging it in, you'll need to make a few modifications to your Pi's network configuration:

sudo nano /etc/network/interfaces/

Ensure that it contains the following information:

auto wlan0
allow-hotplug wlan0
iface wlan0 inet manual

If you want to be able to access your Pi from a static IP (very useful for reliable SSH access when it's tied up behind a flatscreen) you'll need to make the following changes:

auto wlan0
allow-hotplug wlan0
iface wlan0 inet static # <-|
address 192.168.1.XXX   #   |
netmask 255.255.255.0   #   |
network 192.168.0.0     #   |
broadcast 192.168.1.255 #   |
gateway 192.168.1.XXX   # <-|

Then run:

wpa_passphrase <SSID> <Passphrase>\
 | sudo tee /etc/wpa_supplicant/wpa_supplicant.conf
sudo ifdown wlan0
sudo ifup wlan0

You may see an error when running ifup but this didn't seem to affect the actual functionality and a quick ping to Google confirmed everything was working fine. At this point I switched to connecting via SSH and controlled the Dashboard Pi from the comfort of my Desk.

3. Update packages

sudo apt-get update && sudo apt-get upgrade -y # Update the Pi

4. Start Browser on Boot

Install x11 server utils and unclutter:

sudo apt-get install x11-xserver-utils unclutter

Install midori (you could also use epiphany, chromium or a host of other browsers):

sudo apt-get install midori

Then run:

sudo nano /etc/xdg/lxsession/LXDE-pi/autostart

Note: Thanks to Simon Vans-Colina who pointed out Midori is no longer the default browser and must be installed

Comment out the following:

# @xscreensaver -no-splash

Add these lines:

# Turn off screensaver
@xset s off

# Turn off power saving
@xset -dpms

# Disable screen blanking
@xset s noblank

# Hide the mouse cursor
@unclutter

Note: Thanks to Tom Judge who pointed out that inline comments were causing issues in xset

Mine looks like this:

@lxpanel --profile LXDE
@pcmanfm --desktop --profile LXDE
# @xscreensaver -no-splash

@xset s off
@xset -dpms
@xset s noblank
@unclutter

Add the following line to automatically load up your dashboard:

@midori -e Fullscreen -a  http://yourdashboard.yoursite.com

I chose Midori as it appears to render all the elements then refresh, rather than (for example) chromium which renders elements one by one on screen.

Enable booting straight to desktop by running

sudo raspi-config

Step Three - Tuck in!

As you've seen, getting metrics and dashboards up in front of the whole company is a relatively simple process and it's super easy to build your own.

As of writing, we currently have around 5 dashboards - 3 on screens and the others used by teams internally, tracking things like outstanding Github Issues, Revenue, Volume, User Sign-ups, Sales Pipelines and more. How we use these and their impact on our business is left for another post.

Hopefully you've found this useful and if you have any questions, feel free to email me: pete@gocardless.com. If you create your own dashboards or come up with any improvements I'd love to hear from you!

What do you think?
Share on Twitter