Originally published 30 June 2009. This article is retained from the tCI archive. Law, policy or practice may have changed since publication.
Publication of the national Place Survey results finally happened last week. So now we know that 29% of the population think they can influence decisions locally. As with all indicators (this is one is NI 4 – but there are lots of others!) isn’t it wonderful to have a number?
The trouble, however, with such numbers – as any market researcher will confirm – is that they are entirely dependent upon the sample, and professionals in this business have excellent and sophisticated methods to ensure that samples are well-designed and as representative as possible. Ultimately though, all you have from a single sample is a one-time snapshot of the perceptions of those who were asked. Next time around, we will have another snapshot – almost certainly of a different sample. In other words – we will have asked other people. Both samples may be structurally identical, but only to the extent of the characteristics that we are measuring. So they may have the same proportion of males to females, young to old, economically active to inactive ... and so forth. But – unless they are massive - they may well be different in respect of characteristics we are not measuring; in this instance where they live or educational attainment, for example.
So when a researcher claims to have interviewed 1,000 people last year and found 30% in favour of something ... and then a different 1,000 this year and found 45% he or she must be a little cautious in interpretation. There may well have been a change in general sentiment, but we can’t be absolutely sure that individuals have necessarily become 15 percentage points more favourably inclined to the proposal. Although it is possible to devise questions that help, it is rare that surveys of this kind can shed much light on why opinion has changed.
Tracking surveys can be the answer. Here we try to interview exactly the same people. In fact, we can go the whole hog and only count the ones who take part in both exercises. Data collected in this way enables us to analyse the extent to which people’s perceptions change, and provide a basis for investigating the reasons why. The truth is that it is often less useful to know the crude numbers than to appreciate how they change. Many marginal changes in the numbers can be explained by slight differences in the samples rather than real shifts in perception.
Tracking surveys have other advantages too:- • People who are regularly asked their views become accustomed to the questions – and give the answers more thought. They become more deliberative • Such surveys have more credibility and look less like a one-off tick-in-the-box • There is scope to feedback. Last year you told us .... and we did this.... Now what do you think...? • It may be cheaper to recruit one sample for three years than to recruit separate samples every year
In the world of surveys, every one of these advantages can also be viewed as a disadvantage – so there will be a judgement as to when tracking surveys are best. However, in more stringent financial times, cost-effectiveness becomes a real issue.
Add to this the need to meet the requirements of the ‘duty to involve’ and we will certainly see resurgence in the use of Citizens Panels, from which specialised samples of service-users can be created. Imagine the effectiveness of 100 residents chosen to be surveyed every six months about their views on issues that are evolving rapidly – waste recycling maybe ... or anti-social behaviour. If absolute figures are what we want, we will worry as to whether the 100 are truly representative or not. But if our focus is on understanding what Managers and elected members can do to make things better, then that absolute figure is less important than the direction in which it moves.
Therefore ask them again ... and again. Second time around, maybe only 90 respond. Count just those. Next time it may be 80; again count just those. We can use weighting to allow for freakish conditions such as all 20 drop-outs being male ... or young ... or urban (Research experts can help here). But of those you do count, we will know that they have truly changed their opinions, and can work to understand why. This – and little else - is the real basis of successful service improvement.
Trigger points
- Do you currently use tracking surveys? If so, is the data used for service improvement?
- If separate samples are designed for each survey, are you missing out on potential cost-savings?
- To see the National Place Survey results click here
- Consider the best use of Tracking Surveys with Citizens Panels at the Institute’s 5th Annual Conference on Citizens & Community Panels
This is the 139th Tuesday Topic; a full list of subjects covered is available for Institute members and is a valuable resource covering so many aspects of consultation and engagement