Self-selected samples - When consultation respondents might be unrepresentative

An experienced consultation officer in the NHS recently could scarcely conceal her delight. The consultation had resulted in a 1% participation rate, regarded by her bosses and many others as being a creditable achievement.

In the cool light of another day, of course it may look differently. Not only does the arithmetic mean 1% is a very small proportion of the population, but it raises the question of what exactly made that 1% take part. After all, these people are normally self-selected. Indeed most consultation respondents have volunteered to give their views.

In market research, it may not always be crucial. But in consultation - where the aim is to influence decisions or activities, and where the opinions of the public are deemed to be worth considering, it can matter rather more. In the case of our colleague working in the NHS, she may be faced with overwhelming opposition from those who have responded, but also aware that the respondent profile might be unrepresentative.

One immediate reaction often is to test the respondent sample to see how it compares with the overall demographic profile. But even if we assume it is spot-on, it does not explain why these people have chosen to take part whereas others have not. One of the most common criticisms of consultation is that more often than not it is a method of stopping things and a tool for objectors and critics. So what should be done?

1. Try to determine the dynamics of self-selection in each case. If we walk down the High Street with a clipboard and ask shoppers whether they would be prepared to answer a few questions – but without telling them the subject-matter, - those who agree to take part may be doing so because they have the time. They become self-selected by convenience. If, on the other hand, we all wear T-shirts saying Have Your Say on the Future of the Local Hospital, we are more likely to attract those who specifically want to share their thoughts about that particular topic. It becomes self-selected by subject. As a crude rule of thumb, the more controversial the issue, the more likely it is that participants will have opted into the process because they have strong feelings on the matter.
2. Select dialogue methods that reduce the self-selected bias. As in the above example, techniques where people are brought into a discussion without necessarily knowing the subject-matter can help. Recruiting members of a Citizen’s Panel or ad hoc focus groups with a view to considering a range of issues works. So can outreach methods, going out into the community and seeking views. The ubiquitous household postal survey is notoriously prone to self-selection.
3. Even in the methods most vulnerable to self-selection, it is possible to include questions that may assist in determining bias when the analysis is done. Asking in-depth questions about user-experience may help determine whether campaigns for or against proposed change stem from detailed knowledge of services delivered or just from a general awareness of the services offered. Several recent consultations on Library services have illustrated this point.
4. It is possible to test the consultation output against a ‘control group’ of the wider population. If we suspect that opinions expressed in the exercise may not be shared by everyone else, it may be possible to take these views and conduct a more systematic dialogue with a structured sample of the same community. This raises the issue of timing, for by undertaking this useful check-and-balance study after the consultation, there is a danger that the arguments have moved on, or that the wider respondent base has been influenced by publicity from the publication of the initial results. Barry Creasy, the Institute’s ‘Lead’ on dialogue methods therefore favours running a structured sample exercise simultaneously alongside the consultation. This results in early warning of any discernible differences in the data.

None of this should undermine the regard that must rightly be accorded to those who respond. Their views have to be taken seriously. Even if parallel exercises result in very different views, they cannot – and should not affect the overall consultation output. But it should influence decision-makers. Theirs is a tough task in any case, and the best advice is for them to take proper account of the arguments advanced – whoever makes the case.

And that is the crucial point. A good consultation performs the function of exposing proposals to the rigours of open debate. If there are weaknesses in a consultor’s case, a well-organised consultation should expose them. So if participants happen to have arrived at the point of debate by being willing – so what. They deserve to have their arguments fully considered.

Yet again we conclude that maybe consultation is more about qualitative analysis of arguments than quantitative counting of heads.

TRIGGER POINTS

• How prone is your organisation to having your consultations subject to large self-selected samples?
• The themes discussed in this Topic are discussed in detail in the Institute training course Better Surveys and Questionnaires, delivered by Barry Creasy, and next run on Wednesday 27th November in Birmingham
• Join 100+ Institute members at the Institute’s 10th Anniversary Conference - “A World that Listens?” in London 22nd October.
• If you found this Topic interesting, you may also find these of some relevance;
• – Tuesday Topic 227: Counting Heads
• – Tuesday Topic 208: The weight of numbers
• – Tuesday Topic 87: The silent majority

This is the 245th Tuesday Topic; a full list of subjects covered is available for Institute members and is a valuable resource covering so many aspects of consultation and engagement