Originally published 25 September 2017. This article is retained from the tCI archive. Law, policy or practice may have changed since publication.
When consultations have to deal with predictions and probabilities…
Most people have heard the quote that ‘If you laid all the economists end to end, they still would not reach a conclusion’ though fewer may realise that it comes from the sardonic wit that was George Bernard Shaw. Over the years it has been used to emphasise how variable are the diverse views of economists and how difficult they find it to agree with each other, let alone the rest of us.
As we enter an uncertain period in politics and economics, those responsible for forecasting have started worrying. That is because of research published a few years ago by Prakash Loungani of the IMF. It suggested that of the last 150 economic recessions he studied in various countries, only two (yes two) had been properly forecast. In his own words, “The record of failure to predict recessions is virtually unblemished”. Somewhat disturbing news!
Many public consultations amount to a storyline that seeks people’s views on a future scenario where the precise circumstances are, by definition unknowable, and the debate is about assumptions, forecasts and risks. The story is If we adopt this policy, this is likely to happen. Those on either side of the divisive EU referendum occasionally claim they were ‘lied to’ but often they are referring not to factual assertions but to predictions of what the opposing sides claimed might occur following the vote in 2016. That often comes down to who you believe, and in politics, our default position is generally to believe those that we think are on our side and disbelieve any opponents. Yet if they were economists, and if we accept Loungani’s research, we would not believe any of them. How much confidence can we therefore have in predictions that form part of a public consultation?
Of course, there is a difference between the infinite complexities of a nations’ entire economy and the operation of a simple public service. For example: What might be the effects of closing a library? To literacy levels? To Educational attainments? To access to education? To community cohesion? We might well be tempted to assume that professionals working at County Hall should have a reasonable understanding of local dynamics and their guesses might prove accurate. So, take another illustration: If we build this by-pass, what will be consequences for the town that is today full of traffic? For air pollution? For Noise levels? For road accident numbers? For local traders? Will the engineers know? Finally, a more complex example. If we halve tuition fees, what might happen? To applications for university places? To the propensity of poor students to go to University? To University funding? Who knows? In all these cases, a consultation will turn on whether stakeholders find the assumptions of the impact assessments credible.
The implications are clear. Forecasting matters. And Impact Assessments are critical. Proposals submitted for public debate need to be scrutinised against a viable set of predictions, and it is here that we find so much to improve. Here are common problems:
• Consultations which say little or nothing about the impact of the proposals being made. • Formulaic impact assessments which are largely meaningless as they are too vague • No explanations of the methodologies used to arrive at outcome forecasts • Even when operational outcomes are reasonably predicted, equality analyses are weak • Single option consultations with no opportunity to consider whether alternative solutions might yield a different set of outcomes
Too many consultors settle upon a fixed view of an option’s predicted outcome without recognising the margin of error. As users of spread betting realise, the world is moving towards probabilistic forecasting. We are no longer told it will rain; we are advised that there is a 60% chance that it will rain. Even the Bank of England’s Monetary Policy Committee now publishes what they call ‘fan charts’ to illustrate the range of probabilities over matters such as inflation or economic growth.
It is time we saw consultations honest enough to say that, there is a certain percentage chance of this outcome, but a risk of a less satisfactory one. Consultees are fed up with being beaten back by public bodies adamant that their plans will have no adverse effects only to find some time afterwards that they did indeed cause precisely the damage that stakeholders closer to the issues always feared. We need better, more evidence-based methodologies that will produce more robust, more defensible judgements of what may happen when change proposals are implemented.
Loungani was asked about the failure to predict bad news. His reply was that forecasters are risk averse. There was little ‘reputational gain’ by successfully predicting a recession, and much to lose by becoming known as a pessimistic foreteller of doom. It sounds suspiciously like clustering around a consensus prediction that will not alarm bosses or commentators, with the obvious risk that we fail to anticipate important consequences of policy or service changes. Public consultations need to resist top-down exhortations to publish safe uncontentious impact assessments and accept that there may be times when the risks are considerably higher.
TRIGGER POINTS
- How robust are your arrangements for preparing and consulting on Impact Assessments?
- The BBC radio 4 Programme In Business covered some of the issues discussed in this Topic on 7th September and was well summarised in a Guardian Article by Adam Shaw
- The importance of introducing probabilistic forecasting into Impact Assessments will be explored in the Institute’s new training course Conducting a Public Consultation
This is the 328th Tuesday Topic; a full list of subjects covered is available for Institute members and is a valuable resource covering so many aspects of consultation and engagement.