Incident response: email reminders incorrectly sent to customers
Yesterday afternoon on the 4th July 2017, between 2pm and 2:35pm BST (British Summer Time), we sent a number of email reminders to end customers (those wanting to pay one of our merchants), asking them to authorise historic direct debit mandates. The emails were sent to individuals who received a request from one of our merchants to set up a mandate, but who never completed the original request.
All our systems remain secure and uncompromised, no third parties were involved, and we have put in place changes to fix the root cause of this issue. We apologise for any inconvenience caused.
What should I do?
What you should do depends on whether you’re an end customer who received an email from us on 4th July or a merchant collecting money through GoCardless.
For end customers
If you haven’t already taken action, please ignore the email you received. We have deactivated the link within it.
If you opened the link and completed the form, and you don’t want to set up payments to the merchant, please contact the merchant or your bank to cancel the Direct Debit mandate, or alternatively contact us if you have any queries.
If your customers were impacted by the issue, we’ll be in touch with more detailed information.
We’ll be letting you know which customers were affected, and what options you have for handling any action taken by your customers in relation to the emails.
As always, our support team is happy to help.
In detail: how did this happen?
On the ‘Customers’ tab of our dashboard, we have a ‘Remind all’ button, which sends a reminder to all your customers who haven’t finished setting up a Direct Debit mandate with you.
As part of our phone support service, our staff can trigger these emails on behalf of a merchant.
The combination of a bug in our dashboard API and the way our dashboard works when staff are acting on behalf of merchants led to reminder emails being sent for all merchants that had pending customers, rather than just the merchant the support agent triggered reminders for.
As soon as we found out about the issue, we halted sending any further emails, triaged the bugs, and began working on our response.
We have fixed the bug, and performed multiple tests to ensure the fix is working as intended.
We want to reiterate our apology for the inconvenience caused by this issue.
As part of our incident response process, we’ll be conducting a detailed investigation into the issue and our response to it. We’ll be focused on finding any improvements we can - in both our systems and processes - that will reduce the likelihood and impact of this kind of issue happening again.
If you have any questions after reading this, please feel free to contact our support team.
New API Version - 2015-04-29
We last released a new version of the GoCardless Pro API back in November. Since then we’ve made countless improvements and recently we've been working on several new features which warrant the release of a new version.
2015-04-29 is being released on Wednesday, along with a corresponding update to our dashboards. For the majority of integrations, the upgrade will be extremely simple (see upgrading).
The new API docs are available at: https://developer.gocardless.com/pro/2015-04-29/
We will continue our support for v2014-11-03 until the 1st of July, 2015.
Visualising GoCardless' UK Growth
The above animation is a time lapse of customers using GoCardless for the first time. It covers the last 3 years and only maps the UK for now, with each red dot representing a new customer joining GoCardless, then becoming a blue dot for the remainder of the clip. It works even better if you view it full-screen.
Tier One Design Mobile App
Here at GoCardless we love developers, especially when they put together something cool with our API.
We've tried to make our API as flexible as possible to encourage new and interesting use cases. Previously that mostly meant web applications, but no longer!
The clever folks at Tier One Design have created an iPhone App which allows your to see a summary of all your GoCardless information on the go. You can keep up to date with all your customers, bills, subscriptions and account information all in one place right on your phone.
We love hearing from people about their experience developing with GoCardless. Have you created something awesome recently, or are you planning to integrate GoCardless with your app? Let us know!
Update (10th July 2013):
Tier One Design have just released a major update: you can now take payments from within the app. Check it out on the App Store.
Cooking up an Office Dashboard Pi
Since GoCardless started hiring a lot of new faces we've been looking for ways to keep everyone in touch with what's going on. One part of the solution has been adding dashboard screens around the office.
Putting together your own metrics dashboard is actually pretty simple and yields a lot of benefits. This post is a full how-to guide for building your own with a Raspberry Pi, an HDTV and a bunch of hackery.
Step One - Buy your ingredients
Raspberry Pi Components
- Raspberry Pi Model B £32.99
- Raspberry Pi Clear Case £5.98
- Memory Card £6.79
- Wireless Adapter £8.36
- HDMI Cable £3.49
- USB Cable £0.99
- Mains Charger (Optional) £3.15
The TV supports a USB connection, so using the USB->Micro USB adapter, we can actually power the Pi without needing any additional wires going to the mains. Combine this with using a Wifi adapter instead of an ethernet cable and you can attach the Raspberry Pi to the back of the TV without any visible wires. Sweet.
For other configurations see here
Step Two - Prepare your filling
For Dashing Dashboards, use Dashing...
There were several options available to us for creating data dashboards but in the end, the one which seemed most flexible whilst remaining easy to implement was Dashing built by the awesome guys at Shopify.
Dashing allows you to very easily create jobs to pull and generate metrics which you can then send to pre-built (or custom) widgets on any number of customized dashboards in real time.
I won't go into more detail on how Dashing actually works, suffice to say it's awesome and you should check it out.
If you want to get hacking quickly, look at the Getting Started guide on the github page.
The one thing we found was missing for us was persistence. For regularly updated metrics, vanilla dashing is great, if you reboot, you see new data in seconds. However for longer-interval updates (CI status, Deployments etc), when screens are turned off or Pis rebooted we ended up with blank dashboards in the interim.
The workaround we devised was very simple and involved setting Dashing's history to a Redis hash substitute instead of a standard ruby hash.
To do this you will need to add redis-objects to your Gemfile:
# Lets us persist data across reboots gem 'redis-objects'
and then in config.ru add:
# Redis URI is stored in the REDISTOGO_URL environment variable # Use Redis for our event history storage # This works because a 'HashKey' object from redis-objects allows # the index access hash[id] and set hash[id] = XYZ that dashing # applies to the history setting to store events redis_uri = URI.parse(ENV["REDISTOGO_URL"]) Redis.current = Redis.new(:host => redis_uri.host, :port => redis_uri.port, :password => redis_uri.password) set :history, Redis::HashKey.new('dashing-hash')
We've recently started using CircleCI for our Continuous Integration and we really wanted a visualization of our CI status where everyone could see it instantly.
Dashing makes it super easy to add new widgets and since CircleCI has an API, it was relatively easy to come up with an integration of the two, resulting in these widgets:
Step Two - Bake your Pi
1. Image the SD card
I was doing all this on OSX so I followed the RPi Easy SD Card Setup guide and installed Raspbian Wheezy.
2. Get connected
The whole idea of this is to have the Raspberry Pi hidden behind the screen so trailing Ethernet cables isn't ideal. Luckily the Pi supports a range of Wifi adapters.
The Edimax wireless adapter I use eats a USB port but since we don't need it anyway that's not a problem. After plugging it in, you'll need to make a few modifications to your Pi's network configuration:
sudo nano /etc/network/interfaces/
Ensure that it contains the following information:
auto wlan0 allow-hotplug wlan0 iface wlan0 inet manual
If you want to be able to access your Pi from a static IP (very useful for reliable SSH access when it's tied up behind a flatscreen) you'll need to make the following changes:
auto wlan0 allow-hotplug wlan0 iface wlan0 inet static # <-| address 192.168.1.XXX # | netmask 255.255.255.0 # | network 192.168.0.0 # | broadcast 192.168.1.255 # | gateway 192.168.1.XXX # <-|
wpa_passphrase <SSID> <Passphrase>\ | sudo tee /etc/wpa_supplicant/wpa_supplicant.conf sudo ifdown wlan0 sudo ifup wlan0
You may see an error when running
ifup but this didn't seem to affect the
actual functionality and a quick ping to Google confirmed everything was working
fine. At this point I switched to connecting via SSH and controlled the
Dashboard Pi from the comfort of my Desk.
3. Update packages
sudo apt-get update && sudo apt-get upgrade -y # Update the Pi
4. Start Browser on Boot
Install x11 server utils and unclutter:
sudo apt-get install x11-xserver-utils unclutter
Install midori (you could also use epiphany, chromium or a host of other browsers):
sudo apt-get install midori
sudo nano /etc/xdg/lxsession/LXDE-pi/autostart
Note: Thanks to Simon Vans-Colina who pointed out Midori is no longer the default browser and must be installed
Comment out the following:
# @xscreensaver -no-splash
Add these lines:
# Turn off screensaver @xset s off # Turn off power saving @xset -dpms # Disable screen blanking @xset s noblank # Hide the mouse cursor @unclutter
Note: Thanks to Tom Judge who pointed out that inline comments were causing issues in xset
Mine looks like this:
@lxpanel --profile LXDE @pcmanfm --desktop --profile LXDE # @xscreensaver -no-splash @xset s off @xset -dpms @xset s noblank @unclutter
Add the following line to automatically load up your dashboard:
@midori -e Fullscreen -a http://yourdashboard.yoursite.com
I chose Midori as it appears to render all the elements then refresh, rather than (for example) chromium which renders elements one by one on screen.
Enable booting straight to desktop by running
Step Three - Tuck in!
As you've seen, getting metrics and dashboards up in front of the whole company is a relatively simple process and it's super easy to build your own.
As of writing, we currently have around 5 dashboards - 3 on screens and the others used by teams internally, tracking things like outstanding Github Issues, Revenue, Volume, User Sign-ups, Sales Pipelines and more. How we use these and their impact on our business is left for another post.
Hopefully you've found this useful and if you have any questions, feel free to email me: firstname.lastname@example.org. If you create your own dashboards or come up with any improvements I'd love to hear from you!