Originally published 13 June 2017. This article is retained from the tCI archive. Law, policy or practice may have changed since publication.
The danger of pushing your data further than it should go …
This is the effect of having the brilliant Ben Goldacre highlight poor data interpretation at our Conference – and having a General Election the following day. We have all spent the weekend listening to academics, politicians, journalists and commentators all trying to explain what happened. It has been like a well-rehearsed case study in the over-interpretation of data.
We are all familiar with the art of political spin, demonstrated at its worst in the minutes after a television debate as supporters of one politician or another explain why their man or woman did best. It just makes most of us feel queasy. More dangerous is the kind of spin that says “30,000 people voted for Mr X, so therefore they are in favour of …. “. Arguably this is what gets politicians into trouble, for the fact that someone voted in a particular way is not the same as knowing why they acted as they did.
It is quite plausible to argue that the reason why the Conservative Party was disappointed that more UKIP voters did not switch their allegiance to support Theresa May’s Hard BREXIT policy – is that maybe many of them never held strong feelings about the European Union. Possibly they voted Leave in last year’s referendum because they were disenchanted with Mr Cameron, or upset about austerity, or maybe unhappy with all politicians – what’s now known as the anti-politics vote. Many of these are reluctant voters anyway, and when they were called upon to enter another polling station, it is perfectly possible that they might have chosen to vote for the most obviously radical, anti-establishment option available to them. In the absence of Lord Buckethead in their own constituency, it is likely that many of them saw Jeremy Corbyn’s Labour Party as fitting the bill nicely.
This is one hypothesis, and time will tell if it is true. But let’s assume it is right, and that pro-BREXIT Ministers made this assumption that almost everyone of the 52% that voted for Leave actually wanted us to depart, even though it is unlikely to be true. Why did it happen? And why, simultaneously might Momentum and those on the left of the Labour Party similarly assume that all those who voted for Jeremy Corbyn’s party support all the key messages of its Manifesto. The answer is that we all tend to believe what we want to believe. The technical term is confirmation bias.
It can be a real issue with public consultations. Too often, those who initiate the exercise have a view of the outcome they would like to see. That is inherent in any exercise that contains the words ‘preferred option’, and there is pressure to show how consultees supported it.
That pressure must be resisted. The data is what it is. Returning for a moment to voting, all the data showed was the vote – nothing more. To discover why people voted a particular way or to assess what issues influenced their behaviour, we must conduct another – a very different exercise. So, a well-facilitated Focus Group can provide us with a different dataset, but there would need to be many of them before we could build a sufficiently robust picture of the electorate as a whole.
With consultation, we have enough experience to know that relying on numbers is rarely wise. Because consultations disproportionately attract objectors rather than supporters of proposals, we have been tending to favour qualitative techniques for some time. But there are pitfalls to beware. Watch out for someone in a deliberative event expressing a view we know that’s close to the consultor’s favoured outcome; be careful it does not get written up more prominently than the contrary view. Ditto in Focus Groups or Public meetings. Take care also not to interpret non-participation as a vote for your standpoint. “The 99% who have not responded to our consultation are, almost certainly happy to let us make the changes.” Is broadly on a par with “The 77% that voted Remain or failed to vote in the Referendum are presumably opposed to leaving the EU.” Both statements are an over-interpretation of the available data.
Sometimes claims come from those who know no better, and who may never have spent time understanding how statistics should or should not be used. But on other occasions, this type of spin comes from those who know precisely how they are playing fast and loose with the data and who disregard evidence in order to pursue a specific policy outcome or achieve some other purpose. We need to help the first category, and we could start by ensuring that all senior local government officers, NHS Managers and elected Councillors receive training in the responsible use of statistics. As for the second group, maybe we need to name and shame them, and hound them out of positions of influence.
TRIGGER POINTS
- Ben Goldacre’s original book Bad Science was published in 2008 by Fourth Estate, and may be obtained from the Institute.
- Do you have data interpretation standards in use in your organisation?
- Have you experienced cases of over-interpretation? The Institute is always interested in case studies that might help us all learn important lessons.
- Data interpretation features in Institute training courses; Better Surveys & Questionnaires is next running 29th November in Birmingham!
This is the 321st Tuesday Topic; a full list of subjects covered is available for Institute members and is a valuable resource covering so many aspects of consultation and engagement. http://tinyurl.com/jermfrh