Scandal, Surveys and Statistics—An Example of the Transformation of Insights

transformation of insights

It’s a rare day when a change in research methods sparks front page headlines, a public outcry and an investigation by the Federal Privacy Commissioner. But it is happening in Canada right now.

Statistics Canada recently found itself in hot water when it announced it was requesting personal banking data for a sample of 500,000 people. StatCan, as the agency is known, are in trouble mainly for the ham-fisted way they announced that they are requiring banks to hand over the information, but this blunder provides a telling glimpse into the way research is transforming. StatCan is abandoning their traditional survey research on household spending in favor of “big data.” Their scramble to defend their move away from surveys says a lot about where the research industry is headed.

StatCan was established in 1918, around the same time as the very first commercial market research departments were being created. It has long been a survey-driven organization. Quite traditional in their methods, StatCan still largely use phone interviews and even maintain a large cadre of in-person interviewers. But the world of insights is changing, and even StatCan is in transition.

As Canada’s chief statistician Anil Arora put it: “Traditional statistics gathering methods are no longer sufficient to accurately measure Canada’s economy and social changes. It is our duty to leverage best methods and sources to provide facts to Canadians….”

In explaining why traditional survey methods are no longer enough, and why banking records are a more accurate measure of household spending, Arora and former chief statistician Ivan Fellini cited three key reasons:

  • Problems with recall and asking questions people struggle to answer accurately;
  • A lack of representativity of the sample, because response rates are low;
  • Old methods of collecting data don’t fit with how people act today.

Not remembering everything accurately

The existing household purchase survey is based on people keeping a diary of everything they spent. That data has always been problematic, in that detailed recall of all purchases made in a day has always been a bit of a fiction. It’s not that people are trying to lie, it’s just that it is very difficult to answer these kinds of questions accurately.

Annie Pettit has written a valuable and very readable book about writing surveys. The title of the book, People Aren’t Robots: A practical guide to the psychology and technique of questionnaire design, speaks volumes. In an email exchange, I asked her about her thoughts on the way surveys are written today. She marveled at how we can expect things of respondents that we can’t do ourselves.

“We forget what we’ve bought, we misremember where we’ve shopped, and we rationalize that we buy things because we need them, not because we want them. But when it comes time to communicate with people participating in our research, we instantly block out our own illogical behaviors, and expect them to remember precise SKUs, stores, dates and times, and so many other meticulous details about every purchase decision they’ve ever made. That is the definition of illogical.”

Mr. Arora said one of the problems with the household survey is that respondents who fill out diaries on their spending may not think about their digital purchases, including automatic subscription services like Netflix. Mr. Arora said StatCan could produce more accurate data on consumer spending by obtaining a sample of Canadian banking records. He’s right.

Not representative because of non-response

Another problem is that people don’t want to take the time to complete this complicated survey. Fellegi told the Globe and Mail newspaper “Basically, we couldn’t trust the data any longer,” because so many people are declining to participate. “It’s a major, major important survey, but it’s so onerous that people are refusing to complete it in increasing numbers. Now, the response rate is not even quite 40 per cent,” Mr. Fellegi said.

“When 60 per cent of the population [asked] doesn’t respond, the data is likely to be very biased, unacceptably biased. So something needed to be done and [obtaining banking records] is the only viable alternative in sight…”

Old methods don’t fit with how people operate today

Part of the reason people are refusing to participate is that the methods that have been used historically—diaries that people fill out by hand—do not fit with today’s computer-driven world. “As we see the pace at which Canadians now operate in the digital world, we don’t want to be sitting there worrying and working with paper diaries and pencils,” Arora said. “What we do is to make sure that Canadians are served with good-quality data so that they’re not having to rely on anecdotes, fake news [and] alternative facts.”

Out with the old, in with the new

The investigation by the Privacy Commissioner may slow StatCan down on this specific file, but it won’t change a fundamental truth: in the case of things like purchase behavior, there are better sources of data than surveys. As insights professionals we need to change with the times and think beyond the survey.

For more on the transformation of insights, and examples of thinking beyond the survey, check out The Insights Revolution: Questioning Everything or contact us.