Device Agnostic Research: Either Adapt Or Have Misleading Information

Device Agnostic

When did you last check your mobile? Are you reading this on a mobile device? Fully 82% of Americans have a smartphone, and they check it an average of 47 times a day, according to Deloitte. Can you think of anything else people do that often, besides breathe? It’s not eating, drinking, stretching or walking about. Mobile devices have inexorably changed how people act and interact.

The use of mobile devices is age-related. Millennials are much more likely to answer a survey on a mobile device—particularly a phone. Mobile usage is lower among older people but, notably, people also start to shift to tablets as their eyes get older.

When 4 in 10 millennials answer surveys on a mobile device, the effects of the interface are something we must pay close attention to. If the technology by which people answer effects the data we collect, are we getting accurate information? What does that mean for how we understand millennials?

How you present the question matters

Some answers we get are surprisingly sensitive to how the question is laid out. Multi-choice grids are a great example of how the question is displayed can have a profound effect on peoples’ answers.

When online research became the default way for people to do surveys, multi-choice grids turned into a very popular question type. They allow you to gather a lot of data using a relatively small amount of screen space, on a desktop or laptop computer. A multi-choice grid would typically ask something like: “which of these brands do you associate with each of these attributes?” There might be 4 or more brands across the top and several attributes down the side.

On a mobile device, you can’t do a grid. There is too little space on the screen. You need to default to a system which shows the attribute (“Greedy”) and the brands (Australia, USA, UK, Canada, or none of these). It will invariably look something like this:

The difference in how the question is asked has a profound effect on how people answer. In this mobile format, people are less overwhelmed by the complexity of the grid. As a result, they tend to select more brands per attribute. We, therefore, see very different results when we compare mobile responders and online responders—even after weighting to control for age differences.

Problems of interpretation

If you took these results at face value, you would think that millennials (who use mobile more) were more interested in the brands—because they selected more attributes. But you’d be completely wrong.

What if you told the brand team to double down on ad spend for millennials because they are obviously more connected to the brand? You’d be giving incorrect advice, simply because the way in which you asked the question did not consider how the question was presented.

Device agnostic design is the future of feedback

We need survey designs that are device agnostic—questions that work equally well on the different methods people use to interact. People are increasingly embracing many ways of connecting with computers. Virtual assistants like Amazon’s Alexa and Google Home are obvious examples of new connection points that are rapidly gaining ground.

What will the future hold? What new ways of communicating will change how we do surveys? What will it mean for the types of questions we ask? You can learn more about this from our whitepaper The Future of Feedback: Consumer Interest in New Ways of Doing Research.

We need to ask questions that will work equally well on a desktop, mobile, virtual assistant, and, in the future, who knows what. What’s certain is that the use of interfaces will continue to fragment, with important generational differences.

If we stick with the old desktop-driven ways of asking questions, we will gather information which misleads, rather than informs. That’s the antithesis of what an insights department exists to do. We can’t ignore the fact we must change the way we ask questions.

Change is pain. It requires us to explain to stakeholders why we can’t do what we’ve always done. But in the words of U.S. General Ken Shinseki: “If you dislike change, you’re going to dislike irrelevance even more.”

To learn more about device agnostic research design, and how it can help you deliver better business solutions contact us.