On the surface, survey research seems simple. You ask a question and someone answers. Done. Right?
But the reality is a lot more complex than that. The way you ask a question can have a profound influence on the answer, so careful consideration is needed. And sometimes the question you ask will not give you the answer you are seeking. Let’s unpack three lessons learned about the art of asking questions:
1. The way you ask a question can determine the answer
2. Beware of scales that don’t measure what they appear to
3. Asking why won’t get you the answer you need
The way you ask a survey question can determine the answer
The form and wording of the question shapes or determines the answer you get. Questions need to be crafted with a great deal of care. Let me tell you a story about two priests to illustrate this point.
Two older Irish priests were down at the pub having a pint and a smoke. And as they chatted and lit up another cigarette, they discussed smoking and praying. One said to the other “What do ya think Seamus, is it all right to smoke and pray at the same time?” His friend took another puff and exhaling he replied “I don’t rightly know Joe. I suppose we should sort that out.” So, after another pint and much discussion, they decided they’d each write to the Pope and ask him.
Father Joe wrote to the Pope and he said, “Is it permissible to smoke while praying?” And the Pope replied “When in prayer your focus should be on God, and smoking would distract you from that. So, no, I do not recommend smoking during prayer.”
Father Seamus also wrote to the Pope and he asked, “Is it permissible to pray while smoking?” And the Pope replied, “The Apostle Paul told us ‘Rejoice in hope, be patient in tribulation, be constant in prayer’ (Romans 12:12). Of course it permissible to pray while smoking.”
In Dublin, even to this day, Father Joe does not smoke while praying, and Father Seamus prays while smoking: each having gotten the answer to the question they asked.
Beware of scales that don’t measure what they appear to
It is very common for researchers, trying to measure people’s attitudes, to use survey question design that allows people to say they “neither agree nor disagree.” This, unfortunately, renders the statement useless. The problem is a variation on the conundrum of comparing apples and oranges.
When we ask survey respondents how much they agree or disagree with a statement, we are asking them where they stand on a continuum. The continuum can be sliced any way; a four-point scale or one-hundred point scale—it hardly matters—so long as we are looking at where on the spectrum respondent stands. When we throw in a “neither agree nor disagree” option, you allow the respondent to step off the continuum and tumble into a completely different kettle of fish.
And we can’t compare apples and fish.
We then run into serious problems if we want to use the resulting attitudes to create a segmentation, or for modeling drivers of choice, or just about anything that isn’t purely descriptive. We are assuming linearity that everyone’s answers are somewhere along a continuum. But when you have an answer that is off the continuum, you’ve gutted that assumption.
If you were making apple sauce, would you throw in a fish? I doubt it. It shouldn’t happen with agree/disagree statements in a survey either.
“Neither agree nor disagree” is also a vacuously vague response. What does it tell you? Nothing, except that the respondent chose not to stay on the continuum.
There are statements that some people might not be able to agree or disagree with. “I have stopped setting fire to myself” is, for example, the kind of statement that not everyone can agree or disagree with.
If you think that some of the statements you are using will be hard for some people to agree or disagree with, then you are much better off adding options like “don’t know/unsure” or “rather not answer.” At least then you know what you are measuring.
Asking “why” won’t get you the answer you need
“Why?” That’s the question we market researchers want the answer to all the time. But when we ask it, we never get the truth we’re looking for. What we get is a response that is driven by the social context in which we ask it. In other words, people give us the answers they think we are looking for, not the real underlying reason.
Eminent sociologist and social historian Charles Tilly wrote a classic book entitled “Why? What Happens When People Give Reasons…and Why”. In it, he explains: “people do not give themselves or others reasons because of some universal craving for truth or coherence. They often settle for reasons that are superficial, contradictory, dishonest or—at least from an observer’s point of view—farfetched. Whatever else they are doing when they give reasons, people are clearly negotiating their social lives. They are saying something about relations between themselves and those who hear their reasons.”
“Why” seems like such an easy question to ask. But we all know that the answers we get from asking such an open-ended question are never really satisfactory. “Why is that?” you might well ask.
In addition to the fact respondents tell us what we think we want to hear (or what they want to tell us) the reality is we are often incapable of being conscious of why we do what we do. Neuroscientist V.S. Ramachandran summed it up nicely: “Our mental life is governed mainly by a cauldron of emotions, motives and desires which we are barely conscious of, and what we call our conscious life is usually an elaborate post-hoc rationalization of things we really do for other reasons.”
That’s why we at Maru/Matchbox encourage measuring “why” through derived means, rather than asking why. By derived we mean measuring behavior and perceptions in our survey design (like brand image or attitudes toward a product or service) and then looking at how the two overlap. For instance, if we find that “a sexy brand” is very strongly correlated with usage of that brand, we can be sure that people are using it because it makes them feel sexy—even though that would probably not be a reason they would tell us if we asked why.
The bottom line: Don’t ask why. The answer you’ll get is not the answer you’re looking for. Measure “why” indirectly and you’ll get closer to the truth.
Do you have questions about asking questions?
These lessons we have learned from years of tweaking our research methods make it clear that uncovering true insight is not quite as simple as might appear at first. To learn more about some of the best ways to uncover the “why” without asking, read our blog on creative qualitative research methods.
To learn more about how Maru/Matchbox can help you get answers you can act on, contact us.