Some thoughts on the validity of questions raised by the Judicial Review outcome of Claire Stephenson v Secretary of State for Housing and Communities and Local Government.

On 8 March 2019, tCI produced a Briefing Note commentary on the judgment handed down by Mr Justice Dove on 6th March 2019 with respect to the above case. The Briefing Note’s conclusions were five-fold: about scope; about the need for care in review-of-document consultations; about the need for care in consulting when it involves information produced by others; about the importance of considering the ‘reasonable reader’ test; about the need for meticulous data analysis and reporting.

At the heart of many of these points are issues of validity. Validity, when used in the context of collecting data, has a precise meaning. The late Professor Alan Bryman, an acknowledged leader in the field of social research methods defines (in his seminal and regularly re-published textbook Social Research Methods) validity as:

“A concern with the integrity of the conclusions that are generated from a piece of research”

In simple language, we can put this as a question: when you are conducting research on a group of people, how close does what you find out resemble the real situation? We can see straight away that one of the areas we must look at to ensure that our validity is high (i.e. that what we find out really represents the actuality) is how the questions are written – that they are written so that the people reading them understand exactly what it is you are trying to find out from them.

When I am teaching my course Better Surveys and Questionnaires I explain this using the concept of ‘Barry’s World’ – and how to get out of it. Barry’s World is the world I live in; it’s the world inside my head; it’s the ‘real world’ as Barry understands it. I have shorthand for phenomena; I understand how things link up and I see them in the contexts that I have experienced; I know what certain initialisms and jargon words mean. The problem is that Barry’s World isn’t quite the same as the worlds of others, and when I talk to others, concepts that are clear in my head aren’t necessarily seen in the same way by those other people – because they don’t necessarily have the same information as I do, or see the subject in the same context as I do.

It’s a well-known phenomenon in the world of communication, and one that bedevils us in an increasingly internet-mediated, text-based world, where we are frustrated by instructions written by someone who knows what they’re talking about, but isn’t able to see things from the point of the person to whom they’re providing instructions. It’s the equivalent of seeking directions in a strange place, and a local tells you “you go up the road until you come to Bill Smith’s house, then you turn left and keep straight on until you get to the pub that caught fire last year …”. Without being able to inhabit this person’s world, you’re lost. True communication isn’t simply delivering the message, but is when the person to whom it’s delivered understands it.

And so it is with the questions we ask. We may understand perfectly what we mean by a question, but unless we consider how deeply ingrained this question is in ‘our world’ we cannot hope to get a valid response. Although this includes the obvious – such as avoiding jargon or in-house initialisms – it also includes how we, the consultors, see things. So, for example, a hospital administrator may understand clearly the difference between a consultant surgeon and a GP, but a patient may simply see them both as ‘doctors’, and to ask a question that depends on a patient knowing the difference (for example, asking about an experience with a doctor, assuming that everyone knows that this category just means GPs) may lead to misleading data. And the point here is that the person who analyses the data may not even realise that it is misleading. So, the consultor asked a question in their context (a ‘doctor’ is a GP), the patient has answered it in their context (a ‘doctor’ is a medical professional they see in connection with their condition), and an answer is obtained that seems right, but because of the misunderstanding, it’s actually not the answer that was sought – and at this point, validity is compromised.

In the course of my work with tCI, I see many consultation documents and consultation questions, and I would say that the most common problems with them occur because of this lack of attention to the ‘worlds of the consultees’, to addressing the validity of likely responses. This comes in the form of jargon or initialisms, but also, often, in the form described above: a disjunct between the way the person writing the documents sees the world, and the way the respondents might do.

And so it was with the Stephenson/MHCLG fracking case. The civil servants at the Ministry for Housing and Communities and Local Government believed that, when they were mounting their consultation, the broad questions they were asking (such as “Do you have any comments on the changes of policy in chapter 17, or in any other aspects of the text of this chapter?”) were questions about process – i.e. “we’ve taken this text from another document and inserted it into this one; have we done it correctly?”. Of course they thought this – they had spent months in ‘their world’ simply tidying up the National Planning Policy Framework by inserting chapters into it that hitherto had been a series of one-off messy additional documents. What hadn’t occurred to them was that respondents might want to comment on the content of one of these ‘messy additions’. That the chapter on fracking (and the history of controversy behind it) belonged to another ministry didn’t help matters – they were simply charged with inserting it into a more rationalised document, and their world was so full of their task that, when they went out to consultation, they assumed that the consultees would automatically understand that the questions they were asking were about the rationalisation work they had just done.

Not so. And the result, of course, was a deluge of responses from supporters of the campaign group Talk Fracking who wished to re-visit the controversial content of the paragraphs on fracking. The subsequent effective dismissal by MHCLG of these responses (on the grounds that they weren’t relevant to what was being asked in the consultation) was one of the grounds on which Talk Fracking took the Ministry to Judicial Review, claiming that the fourth Gunning Principle (a requirement for the consultor to demonstrate that they had given the output of the consultation conscientious consideration) had been breached. Mr Justice Dove agreed with Talk Fracking and declared the consultation unlawful, invoking the concept of ‘the reasonable reader’, and what such a person might infer about what they were being asked in the consultation questions. He states in his judgment, with respect to both text of the consultation document and the questions asked (highlights mine):

“That specific text provides no suggestion that the substance of paragraph 204(a) is outside the scope of the consultation, or that commentary upon it would be irrelevant.”

and

“The text of question 37 itself makes clear that all aspects of paragraph 204(a) are within the scope of the consultation and matters about which the Defendant wished to receive views in order to inform his proposals.”

Here, then, is a classic case of the civil servants preparing this consultation not getting out of ‘Barry’s World’.

So how do we try to ensure validity is as high as possible? In my questionnaire course I recommend that before finger is put to keyboard when writing questions, consulting organisations should do some initial qualitative spadework with groups of potential respondents: focus groups; meetings; citizens’ panels and so on. A consultor should ‘get into the respondents’ worlds’; hear how they express things; get a feeling for the way they understand the issues (and, coincidentally, discover areas of particular concern so that broad questions can be tightened up or closed down). In consultations that are preceded by engagement and/or options development/appraisal that include groups of people most likely to be affected by proposals, this will happen naturally. At the very least, a consultation document and its associated questions should be piloted properly (allowing respondents to explain what they think they’re being asked, and the reasons for their answers).

We do not know whether the MHCLG undertook any work of this kind on their National Planning Policy Framework consultation documents before they went live (and surely it must have occurred to someone that fracking might be a controversial subject), but, given the arguments advanced at the Judicial Review, it seems unlikely. A lesson for all consulting organisations to consider, even for what may seem like straightforward ‘housekeeping’ consultations.