Over my many years of helping organisations to improve their surveys/questionnaires, one problem area has always stood out. When I teach this in my courses, I use my own name as an exemplar, but it applies to everyone: the questions are so often written from the world-view of the person who is writing them, and this can cause confusion for those responding. It’s so easy to do, and we’re all guilty – we form a view of how things around us work and connect, and we forget that everyone else doesn’t have the same view. There’s a technical word for this: validity – a word that Andrew Bryman describes in his seminal textbook Social Research Methods as “… a concern with the integrity of the conclusions that are generated from a piece of research”. Put in a longer way, this means “making sure that what you get back from your research represents the real picture; the way your research instruments are written and structured shouldn’t interfere with this or skew it”.

Making sure that a questionnaire has a high validity involves work, and it’s another reason why survey methods should not always be seen as quick and easy.

It’s also easy to think that this is just about language, and to check for jargon or acronyms that are commonly used in the organisation that respondents might not necessarily know. But it goes further than that, and is about the way the intent of the questionnaire can get snarled up in the way that the person who writes it sees the world.

Consider the following question:

When you attended the clinic, which type of clinician treated you?

o FY1

o FY2

o SpR

o SHO

Firstly, we should consider the jargon/initialisms, and whether the respondent will know that ‘SHO’ stands for Senior House Officer, but, more importantly, would the respondent know what grade their clinician was? When most people attend a clinic, they ‘see a doctor’ (or a nurse), and that’s about as far as the expertise of the average layperson not familiar with NHS clinician grades will probably stretch. You can see how this came about: the person writing the questionnaire is probably looking at the confidence patients have in their clinician, and trying to link it to that clinician’s grade. In their world, every clinician has a grade attached – they aren’t just a ‘doctor’; but a patient won’t see it like that, and may answer the question randomly (especially if it’s a forced answer question in an online survey), leave it blank, or (at worst) abandon the survey entirely, giving rise to ‘dirty’ or missing data. And this is one of the reasons why, when I teach my course on surveys and questionnaires, my advice is always: don’t start by writing the questions; set out what it is you want to find out first, and only then consider how best to collect it – which will then lead to appropriate questions.

So, how do we cure this? How do we get out of ‘Barry’s World’ when writing our questionnaires? Here are a few suggestions:

1. Don’t write questionnaires on your own; always involve others – preferably those who don’t work with you all the time (and have the same mindset).
2. Always trial/pilot questionnaires with a group of ‘typical’ respondents, and get proper feedback from them (don’t get them just to answer the questions, but ask them to send you a commentary of which bits they found easy/difficult to understand, and why they gave the answers they did).
3. Draw on what you have learned from pre-consultation engagement activities connected with the exercise (or create some). All of that qualitative, face-to-face engagement will give you a very good idea of how potential respondents see the topic, and the words they use to describe it; take note of this for when you write your surveys. Remember, though, that there are not just two world-views (yours and theirs); different groups of people may see things in different ways, so be prepared to write your survey instrument to take account of this.