Hacking on Side Projects: The Pool Ball Tracker

By day, we're a London-based start-up that spends most of our time making payments simple so that merchants can collect money from their customers online. Occasionally, however, we enjoy hacking on side projects as a way of winding down while continuing to build stuff as a team.

Since we have a pool table at our office, we decided to build a system to automatically score pool games. This post focusses on how we approached the initial version of the ball tracker. It's by no means complete, but it demonstrates the progress we made on it during the first 48 hour hackathon.

The balls would be tracked via a webcam mounted above the pool table (duct taped to the ceiling).

We split the system into three components:

• Ball tracker: this reads the webcam feed (illustrated below), and tracks the positions of the balls.
• Rules engine: accepts the ball positions as input, and applies rules to keep track of the score.
• Web frontend: a web-based interface that shows the state of the game.

The Setup

• Refurbished pub pool table
• Set of pool balls (red and yellow)
• Consumer webcam

The camera was set up directly above the centre of the pool table to avoid spending time fighting with projective transformations.

We chose to write the ball detector and tracker in C, using OpenCV. In retrospect, it may have made more sense to prototype the system in Python first. However, C is a language most people are comfortable with, and many of the online OpenCV resources cover the C API.

The Approach

We spent some time thinking about different approaches to tracking balls. There are a few main steps to the tracking process:

• Filter the image based on the balls' colours, to consider only the relevant parts of the image.
• Find objects that looked roughly ball-shaped.
• Use knowledge of previous ball positions to reduce noise and filter out anomalies.

Colour Range Extraction

We converted the input frames to the HSV colour space, which made selecting areas based on a given hue easier. The image could then be filtered using cvInRangeS, which makes it possible to find pixels that lie between two HSV values. We ran multiple passes of this process - once for each of the ball colours - yellow, red, black, and white.

Finding the Balls

Our initial stab involved using the Hough transform (cvHoughCircles) to locate the circular balls. After spending some time tweaking parameters, we got some promising results. 

Tracking Moving Balls

One problem that became immediately apparent was that the tracked balls would frequently pop in and out of existence. One cause for this was the Hough transform failing to handle the deformation of the balls in motion (caused by a relatively slow shutter speed) The colour mask would also occasionally hide balls due to changes in lighting. We needed some kind of tracking.

The first approach was the simplest thing we could do. The position of the balls was stored in memory, and if a pool ball was detected within a threshold it would add confidence to this position. Positions that hasn't been detected for a set number frames were discarded.

Later, we expanded on this approach and mapped the balls onto previous positions with a simple distance heuristic. This meant they would more smoothly track across the table instead of leaving 'ghosts'. This approach can potentially be expanded in interesting ways - for example, using a basic physics simulation to predict where the ball should be based on its past trajectory.

Balls Ain't Round

The approach so far was working for simple cases, where balls didn't touch each other and were sitting still. However, as soon as balls started to move they transformed from sharp, bright u-circles to blurry, elongated blobs. This made them very hard to track using the Hough Transform.

We reimplemented the ball detection code using a generic blob detector, and a bit of morphology. A great deal of parameter tweaking was necessary before we started getting convincing results. In the end, the blob tracker performed much better than the Hough transform did, especially when it came to fast moving balls.

This video shows the ball tracking progress at the end of the hackathon:

Next Steps

Although we didn't get perfect results, we were happy with the progress we made. But this isn't the end - we plan to continue working on it. The main priorities are:

• Speed. The tracker currently runs at about 10 frames / second, which isn't nearly fast enough. We're currently experimenting with moving parts of it to the GPU.
• **Frame rate. **The new GoPro sports camera streams high-definition video at 60fps. This should make it easier to track moving balls between frames.
• More advanced tracking. The motion tracking we're currently doing is very naïve. We've been discussing how we could use more intelligent approaches to compensate for more of the errors from the detection phase.

This is still a work in progress, so if you're interested in helping out or have any advice to offer us, drop us an email or a tweet. We also had help from the London Ruby community - thanks in particular to Riccardo Cambiassi on this project.

In a future post we'll talk about the other parts of the system - the rules engine and the web interface.

If you find problems like this interesting, get in touch - we're hiring.

Discuss this post on Hacker News