Measuring the impact of the BBC’s international charity

Last week at the Media Corporate Social Responsibility Forum, I delivered a speech about how BBC Media Action measures the impact of our media output. I wanted to share some of the points I raised in that presentation in a blog post.

The Forum’s 2013 Mirrors or Movers report explored that a "unique contribution of the media sector [is] that of shaping public debate, changing behaviours and promoting sustainable lifestyles."

BBC Media Action actively sets out to enable change. The BBC itself has a set of public purposes as defined in its charter including sustaining citizenship and civil society; promoting education and learning; bringing the world to the UK and the UK to the world.

As the BBC’s international development charity, BBC Media Action goes a step further. We work on specific development projects while abiding by the BBC’s values and editorial standards. We also use tried and tested processes to measure the impact of those projects.

It is not easy. The audience we want to reach is difficult to survey – often the poorest in society, who are isolated, vulnerable, illiterate, hard to reach. It can be difficult to determine what media output a person has seen let alone isolating the impact of a specific media intervention.

We have surveyed people in conflict-torn Somalia, rural China and the mountains in Pakistan. Last year, more than 60,000 people took part in our research through quantitative surveys, qualitative focus group discussions, expert interviews or other research techniques.

Improving health

One of the areas identified in the Mirrors or Movers report was the role of the media in normalising behaviours - introducing or legitimising new forms of behaviour. We do a lot of this in our health work to change knowledge, attitudes and behaviour.

Research looked firstly at whether the campaign had encouraged conversation about condoms and secondly whether it had changed public perception of condom use in India.

5,718 half hour interviews amongst the target audience of men aged 15 - 49 tested recall and engagement with the adverts. Had they discussed 'Condom Condom'? And what had they discussed?

Then we looked at whether knowledge of, attitude towards and actual use of condoms were higher amongst people who had watched the announcements than among those who had not. 150 million Indian men watched the adverts and the research showed that the more frequently people watched the adverts, the better quality their subsequent discussion about condoms was – and the more likely they were to use condoms themselves.

What’s more, during the ‘Condom Condom’ campaign there was an 8% increase in condom sales across India. It’s perhaps worth mentioning that the adverts worked creatively too, winning CNN's Public Service Announcement of the Year Award in 2010 and a Bronze Palm at the Cannes Film Festival.

Encouraging debate

The Mirrors or Movers report also highlighted the media’s role in shaping public debate. While we don’t see this as something that we should do ourselves, we do invest heavily in creating platforms where people can do so.

A nationally-representative survey of 4,000 people aged between 15 and 65 showed that 32% of people had watched or listened to Sajha Sawal - almost 6 million people. More than half (59%) had discussed the programme with others - a useful indication for us that they had really engaged with it. 89% viewers felt that Sajha Sawal was effective in asking the questions that people would like to ask politicians. There is, of course, a problem with self-reported views of this kind as respondents may want to be 'nice', but it still gives us a useful perspective.

On the qualitative side, activity included focus group discussions with listeners or viewers and, as a control group, with non-listeners or non-viewers; in-depth interviews with public figures, NGOs, religious leaders; and focus groups with staff of partner radio stations.

Qualitative research often gives richer insights. One female listener from the rural area of Surkhet told us, "When I see ordinary people talk in the programme I feel motivated and it gives me confidence that I can also speak in front of the public."

Findings from the focus groups indicated that audiences appreciated the programme’s inclusiveness while in-depth interviews with media commentators said that the quality of the questioning had improved over the duration of the show as members of the audience learned how to better ask questions.

So, we knew that people watched the programme, enjoyed it and said that it was helping hold leaders to account. But was it leading to changes in attitudes and behaviour?

We found that a higher proportion of those who watched or listened to the programme were willing to take part in public action (eg signing a petition or participating in a demonstration) than those unexposed. And they were more likely to vote in the next general elections (69%) compared to the unexposed group (59%).

But this begged the question of cause and effect. Was it the more educated and politically engaged that watched or listened to the programme to begin with or could the higher level of political engagement really be attributed to Sajha Sawal?

Our research staff use statistical analysis to rule out other factors and have been able to show that there is a clear link between exposure to our programming and political participation, even when other possible influences are taken into account (such as gender, age and education). And we were also able to see that those who watched the programme more often were more likely to become politically engaged. (For more on this work, read the research team’s fascinating report.)

BBC Media Action has grown substantially over the last few years and a great deal of that growth is down to the investment we have made in measuring impact. Donors trust us to do what we say we are going to do, and when we have done it to prove it through research.

We are a media charity that exists to effect change. It is our research and measurement that underpins our ability to achieve change and makes us not just a mirror but, we believe, a real mover in achieving development outcomes.

Caroline Nursey is Executive Director, BBC Media Action

• Today it was announced that Caroline Nursey has been awarded Order of the British Empire.
• Read other blogs about BBC Media Action
• BBC Media Action also blog regularly over on their dedicated blog.