Testing BBC iPlayer on Chromecast

Hello, I am David Colter, part of the team that brought BBC iPlayer to Chromecast on mobile devices. I am a Senior Test Engineer at BBC North in Salford Quays where a great deal of our work on mobile devices is done.

We developed our Chromecast capability in partnership with Google. They provided us with their software development kit for us to integrate Chromecast connectivity into BBC iPlayer apps. We were also given some design guidelines to define how the cast experience was to work in use. We did this on the Google Android and Apple iOS platforms side-by-side, for phone and tablet form factors. No platform was allowed to race ahead of the other; nothing was declared to be done until it was working and tested on both platforms.

Our Core Engineering team operates on agile behavior-driven development principles. The functionality of the app is broken down into small user stories which describe a small part of the journey through the app towards the desired result. These stories are written by stakeholders from product, development and test all working together. Once a common understanding has been reached, the stories are prioritized and then put into a sprint, which is a two week long effort that gets the stories developed as software, tested, fixed where necessary and declared done.

At the end of each sprint, we demonstrate our completed stories to our Product Owner and any other interested stakeholders including, in this case, Google. In this way they can see what we have done and gain confidence in the progress of the project. It gives them an opportunity to change direction if they decide that what we are producing does not match what they intended. They might also want to change direction due to external influences that have developed while we have been working on the project. At the end of each sprint, our team also holds a retrospective to discuss what went well and what did not, and this guides the direction of the next sprint.

Testing is an integral part of the lifecycle of a user story. We sit together with the developers, attend the same morning standup meetings and track our work as another stage in the same process. Once a story has been developed in software it is passed to a tester who immediately sets to work. First we make sure it passes all its acceptance criteria, seeing that it does everything it is supposed to do. Then we crack our knuckles, put on our black hats and try to break it.

If we find a defect or some questionable behaviour, the first action is to check with the developer, who is only a scoot of the chair away. Very often they can just facepalm and fix the problem straight away. If a defect is likely to have a longer lifespan than this, we will raise a bug report describing exactly how to reproduce the problem and what the expected versus actual behaviour is. These are then prioritized and worked on in exactly the same way as the rest of the user stories.

In this project we came up with the idea of portable network environments for developing and testing in. We used tiny Raspberry Pi computers as Wi-Fi access points. These were set up to filter and intercept some of our web traffic, replacing it with local responses specific to development or testing. With this we were able to keep development and test within separate environments while still using all of the real-world content available on BBC iPlayer. A Raspberry also became our travelling demo environment, enabling us to show the current state of the project to senior stakeholders wherever they were.

One of the challenges that affects mobile testing is the wide range of devices that our apps run on. We select a representative sample of phones and tablets, weighted by the statistics we gather on the current mobile usage of iPlayer. We will never have the time or budget to test all features on all devices. We try to get the best coverage for the greatest number of users without ignoring older or less popular devices. It is always a difficult decision and only time, feedback and comments from you, will tell us whether we have got it right this time.