The world of market research is at a time of transition. It is sure to change, but the question is how? To understand the future, you need to know your past. The insights industry has processes, customs, and assumptions that history and tradition have bequeathed us. Some of these will be enduring qualities. And some will be anchors that could snare us and tether us in place just as the flood waters of change rise and drown us.
Let’s look at the forces that have shaped the insights industry today, so that we can be ready for tomorrow.
The roots of market research
Surveys are so common (“On a scale of 0 to 10, how likely are you to recommend this book to a friend or a colleague?”) that it is hard to imagine a time when they did not exist. But, indeed, what we call the market research industry is less than 100 years old—a mere blip in humanity’s journey.
The direct antecedent of today’s survey research industry started in the 1920s, but it came to prominence when George Gallup used sampling theory to gather results that enabled him to correctly predict the 1936 presidential election. At the same time, Literary Digest’s “straw poll” got the election results spectacularly wrong. This made the importance of good sampling crystal clear.
This triumph of “scientific” surveying traces back to three earlier developments: social research, psychometrics, and sampling theory.
Social research efforts emerged at the end of the 19th century. Famous early studies included Charles Booth’s Life and Labour of the People in London—an in-depth look at the social conditions of the poor—and the Hull House Maps and Papers of 1895—a study of poverty in a portion of Chicago. These studies were not the kind of very narrow survey research we do today. They were more sprawling, omnivorous investigations.
“In Booth’s time, the survey did not specify or imply specific modes or instruments of data gathering, such as interviews or questionnaires,” writes Jean Converse in her excellent Survey Research in the United States: Roots and Emergence 1890-1960. “Indeed, as we have seen, the early social surveys used a melange of techniques—the more the better—to gather data: questionnaire, interview, letter, direct observation, participant observation, systematic counts of observed behaviors, physical examination and measurement of houses and human beings; family budgets of income and expenditure; and aggregate data on population, migration, births and deaths, health and disease, wages, and prices.”
Booth’s investigation inspired people because it revealed that which was ill-understood and painted a compelling and unsettling portrait of life in London. It did what research does best: expose the richness of reality.
Psychometrics and the measurement of attitudes
Booth’s enthusiastic investigations were deep and rich, but they did not embrace the kind of formal quantitative methods of asking questions that are so commonplace today. More focused and quantitative surveys sprang largely from the development of psychological testing.The first modern intelligence test was developed in 1904 by Alfred Binet and Theodore Simon. They were commissioned by the French Ministry of Education to devise a test that would distinguish “mentally retarded” children from normally intelligent but lazy children. At the same time, others were developing psychological measures that aimed at providing a precise account of belief and behavior.
James McKeen Cattell, in his classic 1895 paper Mental Tests and Measurements, set the groundwork when he wrote, “Psychology cannot attain the certainty and exactness of the physical sciences, unless it rests on a foundation of experiment and measurement. A step in this direction could be made by applying a series of mental tests and measurements to a large number of individuals.”
The study of attitudes and their measurement began to flourish in academic social psychology in the early 1920s. If we want to reflect on why we use the measures we do, it is crucial to understand that the psychologists who developed these measures worked primarily with their university students. Their interest was in developing academic measures of complex issues, and students were ideal.
According to Converse, “First, one needed people with some talent for attitudes—literate, comprehending, articulate, and self-conscious to some extent about their intellectual, political, and moral positions; people, in sum, who were trained in having attitudes . . . . Students also did not pose problems of academic translation—that is, they did not require the simplified wordings, less abstract ideas, and concerns and situations that were closer to common experiences among the broad public. Second, one needed people with the time and tolerance, and students could be—gently—imposed upon.”
Does that sound like the person doing your survey, who was attracted by an ad offering to “put cash back into your wallet”? Do they have “time and tolerance”?
This has important implications for us, because many of the types of complex scales and measures we use trace back to the work of these psychologists. We inherited a template for how to ask questions that has very little to do with the world we live in today. To learn more about a better way forward download our whitepaper Changing Times Demand Rethinking Old Approaches: a case for quick, reliable and easy-to-answer questions.
There are not too many people who get excited about sampling. But representative and reliable sample is the backbone of survey research. Without it, the whole enterprise will fail. Bad sample means incorrect results, leading to wrong decisions. With bad sample, we imperil the reputation of the entire industry.
The notion of sampling is also relatively young. Anders N. Kiaer, the director of the Norwegian Bureau of Statistics, first proposed sampling at a meeting of the International Statistical Institute (ISI) in 1895. Many people thought that the idea of not using a census to measure the population was heretical. Kiaer was able to show that his methods, as he refined them, produced results that matched the census. But it was 1906 before an Englishman named Sir Arthur L. Bowley introduced the idea of probability theory in sampling at an ISI conference. He was the first to suggest the error in a sample could be measured. And he was so bold as to suggest samples could replace a census.
Polish statistician Jerzy Neyman ushered in the modern era of sampling with his 1934 publication On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection. “Sampling statisticians view the 1930–1940 period as the practical start of their profession,” writes Robert Groves, former Director of the U.S. Census Bureau. “Neyman’s article in 1934 convincingly presented evidence that probability sampling offered bias-free estimates and measurable sampling errors. The early founders of that field in the United States told of the excitement of young statisticians in Ames, Iowa, and Washington, DC, studying his paper, bringing Neyman over to visit, and teaching each other its implications. There is even an oft-repeated story of their using the meeting table in the U.S. Secretary of Agriculture’s office (then a department that was a hotbed of survey development) to meet after the workday to discuss new developments.” The work of Neyman and the U.S. government researchers also shaped the efforts of market research pioneers Gallup, Starch and Roper.
The new scientific approach to sampling gave survey research the representativeness and reliability it needed to have widespread application. Now researchers knew they could ask a properly selected subset of the population a question and get the same answer twice. Without those advances, the business of market research would not exist today.
Good thing Neyman is not alive to see how most sampling is done these days. He would be appalled. Sadly, as an industry, we have largely abandoned what was learned about representative samples. This has been driven by three powerful forces: a decline in trust, a change in how people communicate and—most importantly—an insatiable desire to reduce the cost of doing research.
Learning from our past, to chart a better future
In a time of evolution, repeating history is good, if what you are doing works. But repeating a dysfunctional approach is a sure road to extinction.
When we look back at our heritage, we can identify some ways in which we have moved away from our roots—much to our detriment. In other ways, we have clung to the old ways—retaining methods that grind against the reality of today. We need to be aware of our heritage and think about whether it is helpful or hurtful.
The omnivorous approach of Charles Booth was very fruitful. He collected all sorts of information—including surveys. But it was his synthesis of all those sources that made his endeavors so impactful. We have too often been seduced by the easy answer of the survey. Context is critical. We should embrace many sources of information and let them all shape our perspective. We must return to our heritage.
Psychometricians were the kings of precision when survey research was a fledgling, attempting its first flights. But their influence imprinted on us too deeply. Methods that focused on precision in measuring the attitudes of 20th century university students in a laboratory are unhelpful when it comes to asking 21st century people on their mobile phone what they think about a new package of bacon. We need to rethink our assumptions about what’s a good question.
The early raging debates about the validity of thoughtful approaches to sampling have been tossed aside for savings in cost and an increase in speed. This is insanity. It undermines the very premise of providing information that is true. Without truth, there is no point to an insights industry. Let’s remember how good sampling gave wings to our fledgling endeavor.
We need to change.
Change is good
Let’s leave the last word to Robert Groves, of the U.S. Census Bureau: “Survey research is not dying; it is changing. The self-report sample survey provides insights into the thoughts, aspirations, and behaviors of large populations in ways that data tracking naturally occurring behaviors are unlikely ever to capture. The survey method has strengths and deficits that are reflections of the society that it measures; … [it] is governed by norms that can and do change. Survey research has always and must always adapt to those changes.”
The Insights Revolution is Coming
The Insights Revolution: Questioning Everything
It’s time to start the insights revolution.
Andrew Grenville, Chief Research Officer at Maru/Matchbox says the insights industry is in trouble.
It’s not growing and it does not have real influence in the boardroom. So what to do? This book takes a problem/solution approach that shines an uncomfortable light on familiar practices before suggesting a better way forward.