Is This Water Safe? A Deep Dive into the Reliability and Representativeness of Survey Samples

water hero image

Clean water. Electricity. Traffic lights. These are all things we take for granted, because they are reliable. So reliable that they are boring. They are reliably boring. And that’s a good thing.

The credibility of the market research industry is based on the assumption that the results of our surveys are representative and reliable. But that belief is increasingly wrong.

Sample is now available from a myriad of sources, and it is increasingly inexpensive. But where does it come from? Does the source introduce bias? What are the respondent’s motivations? Can the results be trusted? Are they reproducible? Are they representative?

These are critical questions the industry can’t afford to ignore. In fact, it’s bringing the research industry to a crisis point.

The reality is, a lot of sample sources are unreliable and unrepresentative. They point you to wrong conclusions and lead businesses to make ill-informed decisions.

Market research’s sole function is to provide insights that improve decision making. When the research you are using misleads you, it not only leads to a bad business decisions, it also undermines the whole reason for doing research in the first place.

In the pursuit of inexpensive sample, the industry is shooting itself in the foot. That’s why we have invested in an ongoing series of studies looking at the impact of sample source on data quality.

In the latest installment — Boringly Reliable: Evidence of the consistency of Maru/Matchbox market community sample — we conduct ongoing monitoring of the representativeness and reliability of our market communities. And while the results are about as exciting as saying “yes, this water is clean and good to drink”, it is vital information. It provides powerful evidence of the quality of your data which you can share with internal stakeholders.

One of the newer sources of sample—one that is attracting a lot of attention because of its low cost—is publisher sourced sample. This sample comes from people seeking access to specific gated content on publishers sites. In The High Cost of Cheap Sample we demonstrate how dangerously wrong data from this sample source can be and point out how it can result in very bad business outcomes.

Another increasingly common source of sample is people who are part of specific loyalty programs. In Wine, Cheese, Scotch and Sample we demonstrate how the hidden biases in sample from these kinds of sources can greatly distort results and lead to disastrous decisions.

We look into river sample and the trend toward the hollowing out of panels and the substitution of river sample in Last Call at the Sample Oasis and River, Routers and Reality. Here we demonstrate the detrimental effect this is having on reliability.

With so many problems with much of the sample available today, it’s reassuring to know that some sample sources are like electricity and clean water—boringly reliable.

To learn more about the representativeness and reliability of our market communities download our whitepaper Boringly Reliable: Evidence of the consistency of Maru/Matchbox market community sample.

Download the Whitepaper

Learn About The Power of Omnibus