Worsening problems at the Office of National Statistics (ONS)

I started my research career in the early 2000's conducting telephone voter ID surveys for the Labour Party, using paper call sheets that would be manually entered into databases. I've also knocked on hundreds of doors for council and housing association surveys. This front-line experience gives me particular insight into today's crisis in survey response rates, which extends far beyond just official statistics.

Having experienced the evolution of survey methods firsthand, I've witnessed how gathering reliable data has become increasingly challenging across the entire research industry. What we're seeing at the Office for National Statistics (ONS) is just the most visible example of a broader problem facing all researchers and data gatherers.

The crisis at the ONS has now reached a critical point. The ONS has announced it's scrapping the Survey on Living Conditions (SLC) to divert resources toward fixing its collapsed labour market statistics. Response rates for the crucial Labour Force Survey hit a low of 17.4% last year, making it "essentially useless" according to the Institute for Fiscal Studies. The Bank of England governor himself has expressed concern about this "substantial problem."

The issues stem from both systemic changes - such as the pandemic-forced switch from in-person to telephone interviews - and a growing public reluctance to participate in surveys. This reluctance affects all types of research, from market research to academic studies, making it increasingly difficult to gather representative data.

The human dimension of this crisis shouldn't be overlooked. The ONS's 300 frontline workers, often paid the minimum wage, face the daily challenge of securing responses in an increasingly resistant environment. With staff turnover as high as 25% annually, maintaining experienced interviewers is a constant struggle - a problem familiar to anyone who has worked in field research.

While the ONS is working to address these challenges through a new "transformed Labour Market Survey," this won't be ready until 2027. Sir Ian Diamond, the UK's national statistician, acknowledges they need 1,500 interviewers for "great statistics" but will remain 500 short even after planned recruitment.

The implications extend beyond government statistics. As Adam Corlett from the Resolution Foundation notes, "The challenges of ensuring rising living standards and falling poverty are too great for policymakers to be flying blind with poor data." This applies equally to businesses, charities, and researchers trying to understand and respond to social and economic changes.

The research industry faces a watershed moment. Without addressing the fundamental issues of public engagement and resource allocation, we risk losing our ability to make informed and good strategic decisions.