Resident satisfaction surveys: are yours fit for purpose?

Resident and citizen satisfaction surveys are a standard tool in public sector performance reporting. Used well, they reveal what residents think and why. Used poorly, they produce numbers that mislead decision-makers and fail scrutiny. The difference lies in how surveys are designed, conducted, and reported.

What is a resident satisfaction survey actually measuring?

A satisfaction survey measures an evaluative judgement, not a direct record of events. Satisfaction scores are shaped by prior expectations and personal circumstances, not only by service quality. Treating a score as a direct performance indicator, without understanding these influences, is one of the most common errors in public sector survey practice.

The research literature is clear on the distinction between satisfaction, an overall evaluation; experience, what actually happened; and trust, the broader relationship between residents and institutions. Satisfaction data is most useful when combined with experience-based questions that explain the reasons behind it. The Tenant Satisfaction Measures (TSM) framework, introduced by the Regulator of Social Housing for registered providers in England, reflects this directly: it requires landlords to measure both overall satisfaction and specific service experiences in a way that allows comparison over time. Public bodies running satisfaction surveys for accountability purposes should apply the same discipline.

Why does sampling method affect whether your results can be trusted?

How respondents are selected is one of the most consequential decisions in survey design. Open surveys, where any resident can respond via a website, newsletter, or social media post, are vulnerable to voluntary response bias: people who feel strongly, positively or negatively, are more likely to take part than those with moderate views. The result may not reflect the resident population.

The American Association for Public Opinion Research (AAPOR) has documented that response rate alone is not a reliable indicator of survey quality. A high response rate can still produce biased results if those who responded differ systematically from those who did not. What matters is non-response bias, not non-response itself.

Probability sampling, the practice of drawing respondents from the full resident population using a structured and documented sampling frame, is the preferred approach. The TSM framework makes this explicit: registered providers must demonstrate representativeness, not merely report sample size. A satisfaction survey submitted to a scrutiny committee, or used to justify a service change decision, carries an implicit claim about whose views it represents. If that claim cannot be substantiated, the survey is a liability.

Where do telephone surveys fit in, and what must be managed?

Telephone interviewing remains a legitimate and widely used method for resident satisfaction measurement. Interviewers can clarify questions, reach residents who are not digitally engaged, and manage complex survey routing accurately. It is not a legacy method.

Its limitations are real, however. Contact rates have declined and call screening has increased. There is a recognised risk of social desirability bias: respondents may moderate criticism when speaking directly to an interviewer. Research in the Journal of Survey Statistics and Methodology identifies good practice for managing these risks: multiple call attempts at varying times, active monitoring of demographic representativeness during fieldwork, and transparent reporting of methodology. Mixed-mode surveys, combining telephone with online or postal methods, can improve coverage, but research published by the National Institutes of Health found that adding channels does not automatically eliminate bias. Design logic matters as much as channel choice.

What does a defensible resident satisfaction survey require?

A resident satisfaction survey that can withstand scrutiny, whether from a regulator, a scrutiny committee, or a judicial review, should be able to demonstrate the following six characteristics.

• Probability-based sampling: respondents drawn from the full resident population using a documented sampling frame, not recruited through open invitation.
• Representativeness monitoring: demographic, tenure, and geographic balance checked during fieldwork, not only once results are in.
• Correct questionnaire sequencing: overall satisfaction asked before specific service questions, because preceding questions influence subsequent answers.
• Accessibility by design: routes provided for digitally excluded residents, those whose first language is not English, and those with sensory or other impairments. Coverage error cannot be corrected fully after the fact.
• Independence: where surveys are used for accountability or regulatory reporting, independent data collection strengthens credibility and reduces social desirability effects.
• Transparent reporting: published results should describe the sampling approach, fieldwork timing, survey mode, achieved sample size, and how representativeness was assessed.

Weighting, the statistical adjustment of results to align the respondent profile with the known population, is a useful corrective tool. It is not a substitute for sound fieldwork. Weighting cannot correct for the scenario that poorly designed surveys create: respondents who differ from non-respondents in their actual satisfaction levels.

Resident satisfaction surveys are also a relationship. Asking repeatedly without acting on what residents say reduces future participation and, with it, the representativeness the survey depends on. The organisations whose survey data holds up to scrutiny are usually those whose residents have reason to believe it will be used.

How tCI Can Help

Advice and Guidance
A tCI faculty member will work alongside you to support the development of your decisions and engagement approach. We provide independent, constructive advice at critical stages, helping you strengthen stakeholder mapping, test communication strategies, and plan robust post-decision engagement. Our role is to act as a critical friend, offering practical recommendations grounded in consultation law and good practice that build confidence in your process.

Risk Assessment
Early identification of legal, political or reputational risks in your engagement approach. Using tCI’s five-risk methodology, we spot gaps before challenge arises, helping you strengthen stakeholder communication and demonstrate procedural fairness from the outset.

Executive Briefings
Concise updates for senior leaders on consultation law, engagement duties and post-decision risks. Helps boards and leadership teams make confident, defensible decisions when under pressure, with clear guidance on what good engagement looks like after difficult choices are made.

Whether you’re preparing for a high stakes service change or building defensible evidence for complex decisions, we can help.

Contact tCI: hello@consultationinstitute.org