Hello there my name is David Wynne and I’m a technical architect and founder of Red Badger, a creative software workshop that has been working with the BBC to develop features such as BBC Now.

Through BBC Connected Studio we’ve been working closely with Eleni Sharp, BBC Homepage product manager and her team to design, test and develop the BBC Now pilot and you can read more about the feature in Eleni’s recent blog post.

The Challenge

Our challenge with BBC Now was to capture all the content being published both on bbc.co.uk and on other platforms and to bring it all together into a single unified stream of real-time push updates that are instantly delivered live to the BBC Homepage.

Broadly speaking this broke down into four technical challenges:

Monitoring for new content. Contextualising new content against a BBC brand (such as Top Gear). Pushing content out to the BBC Homepage in real-time. Aggregating common terms within the content to establish trending topics.

The Solution

Each type of datasource (RSS, Twitter, BBC iPlayer content etc) is monitored by a separate Content Processor. Each Content Processor identifies new content, transforms it into a standard structure, associates it to a BBC brand and publishes it to the Firehose.

The Firehose is an unadulterated, unified stream of all new content from every Content Processor. The Firehose allows for multiple subscribers who can use or transform the data and produce their own streams of content.

The primary subscriber to the Firehose is the BBC Homepage which surfaces each item of content as an update in the Fast Lane. The Fast Lane is a column on the Homepage that receives push updates, from the Firehose and displays them in chronological order.

The Trend Analysis module also subscribes to the Firehose and receives the same set of content. The Trend Analysis module extracts common terms from the published content and aggregates that content using those terms, creating categorised ‘buckets’ of content.

When the buckets are sorted the Trend Analysis Module produces a list of the most common terms appearing in content over a given period and facilitates content filtering.

Flow diagram of BBC Now

The Tech

The heart of BBC Now is built using Node.js, a relatively new technology but one which has picked up huge momentum and for good reason. Node.js is a technology born of the web, for the web. It uses Google's V8 JavaScript engine on the server and an event loop to drive code execution. It allows scalable, real-time web applications, quickly.

BBC Now is built out of flat, independent modules making it easy to maintain and scale out. The thin layer of glue that holds them together is Redis. Redis is an advanced key-value store that, like Node, uses an event loop to drive execution and is both fast and scalable.

Redis also offers a fast and scalable implementation of the Publish/Subscribe (or pub/sub) messaging pattern. A pub/sub implementation allows publishers (in this case our Content Processors) to publish data without needing to know how it is delivered to other parts of the application.

Conversely the subscribers (in this case the BBC Homepage & Trend Analysis module) can subscribe to topics of data without needing to know which other part of the application produced it. They simply react to new data as it’s published.

This separation of concerns encourages each part of the application to concentrate on their single responsibility, which in turn means the system as a whole becomes easier to scale and reconfigure.

In essence, BBC Now is all about subscribing to data and publishing it elsewhere making Redis a perfect partner.

In designing the real-time push updates to the BBC homepage we had a few options.

This most traditional method of polling, whereby the client (your browser) periodically asks to the server (the BBC homepage) for updates, seemed like a poor choice. Most of the work we had put in was to convert passive data sources into active ones, returning them to a passive data source seemed to be heading in the wrong direction.

WebSockets, a relatively new technology that allows a constant two-way conversation between client and server (think of an instant message conversation), seemed like an obvious choice. But would have perhaps been a bit overkill given we only needed one-way communication.

Server-sent Events (SSE) provided the perfect lightweight middle ground. SSE are expressly designed for providing push notifications from a server to a browser client and have wide browser support (apart from IETM). SSE operate over HTTP (unlike Websockets) which means that JavaScript polyfills can be employed to bridge the gap for older browsers which don’t natively support the mechanism.

The Future

Besides further feature development (more data sources, personalisation, mobile) we’d love to dovetail our work with that of the BBC Linked Data Platform team which would provide a wealth of contextualised data that would really push us onto the next level.

I’d love to chat more about BBC Now – please leave a comment below or you can catch me on Twitter @dwynne or via the #bbcnow hashtag.

David Wynne is a technical architect and founder of Red Badger.