In today’s newsletter, there is a cautionary tale about allegations of bias when the Government used a specialist Company to help obtain a bigger response to the consultation on a proposed Holocaust memorial next to the Victoria Tower of the Houses of Parliament. The article can be read here. Critics are suggesting that the responses received as a result of the subcontractor‘s activities differed markedly form those received earlier in the consultation period.

Leaving aside the specifics of this particular case – which readers can judge for themselves, it is worth exploring whether consultor organisations might be open to criticism when they subcontract key parts of a consultation to third parties.

Dialogue methodologies have always had the potential to influence the nature of the substantive debate. If the chosen methods are particularly favourable to one group or another, it will obviously bias the responding sample to those people. It is why a portfolio of response methods is always recommended and why external Quality Assurance can be useful to ensure that the mix of methods overall, minimise any risk of favouring one side of an argument over another.

When consultors appoint a specialist consultancy to run their consultations, it would be unusual to give them a totally free hand in selecting these methods; it might be equally unreasonable to stipulate in minute detail what the dialogue programme should be. Somewhere between these two extremes is a position where the parties work together to design a process that balances out the bias inherent in different dialogue methods. Thus an online survey, which will, we suppose, attract a certain kind of demographic, may need to be offset perhaps by a face-to-face street survey which might encounter a different profile of respondent. And meetings will attract another …. And so forth.

So what happens if a terribly well-promoted online campaign run by a third party generates a large volume of responses, mostly supporting one proposal. And more traditional methods, organised in-house, and attracting far fewer participants, happens to result in a majority supporting a different proposal? Would those in the latter category have grounds to complain? And what if the online option had been introduced late in the consultation and the balance of responses suddenly changed as a result of the new initiative?

Geography can also matter. When TfL consulted on a proposed ‘West London Tram’ fifteen years ago, it found that responses gathered from those within the wider catchment area of the new route were mostly positive. But data from those immediately on the route itself, and gathered using a different methodology produced the opposite result; unsurprisingly, those who were most heavily impacted reacted far less favourably. Did they have cause to complain when those further away found it easy to say YES to a less detailed proposition?

These are difficult questions, but maybe we can advance three straightforward items of guidance:

• Try to anticipate the likely bias of each method, particularly if those methods are targeted at different stakeholder groups. Build a comprehensive dialogue programme that balances a method that favours one group against one that favours another. Use stakeholder mapping to check your design.
• Avoid introducing a new initiative mid-way through a consultation – especially if could be construed as biasing the exercise by one side or another of an argument.
• Be rigorous in analysis, ensuring that responses are considered according to the data gathering method, and exploring reasons why some groups will have preferred one proposal to another. Don’t let the quantitative obscure the qualitative …

Finally, there is the advantage of obtaining a thoroughly independent assessment of the entire consultation. That independent element is a requirement of major infrastructure consultations per the Aarhus Convention but is not always undertaken. The Institute, of course offers its own best practice Quality Assurance, one element of which is to sign off the Project plan and looking in depth at the balance of dialogue methods and the way they are to be deployed. It also looks carefully at plans to employ specialist third parties, and we have the skills and experience sometimes to oversee the quality standards they observe, especially when this is a little beyond the consultor’s normal range of expertise.

Those planning a contentious or a large-scale consultation should certainly look at the QA service; it can save money by avoiding less necessary embellishments as well as be an assurance of best practice.

But then we would say that, wouldn’t we?

We’re biased!