Idea Filter: Evidence of a Sensitive and Easy to Answer Approach to Idea and Concept Testing

idea filter

Developing Idea Filter

When developing Idea Filter, our aim was to develop an approach to testing ideas that is:

  • quick and easy to answer;
  • mobile friendly;
  • sensitive to differences between ideas;
  • and leads to the same conclusion as traditional methods.

We conducted a three arm study in which we asked two types of binary questions and compared those results to a traditional multipoint scale. See methodology section for specific questions used.

Comparing a 5-point scale to alternative binary measures

We know that binary measures tend to be much faster to answer than a scaled question (1). We also know that they are easy to answer on a mobile device, so we sought to understand how binary measures compare to the more common scales typically used in testing ideas. (2)

We used five different concepts in the test and each person only evaluated a random three of the five. The concepts ranged from well-known products – M&M Peanut and Hershey’s Cookies ‘n’ crème – to products that exist elsewhere – Appletiser – to products not yet released – Mobile Wallet and Wireless Laptop.

The three types of questions returned different percentages for interest in buying, with selection returning lower numbers and yes/no higher numbers. The pattern, however, was the same for each measure—what was highest or lowest for each type of measure was identical. In fact, the rank order was identical for all five concepts.

purchase intent

The pattern was very similar for the other measures – liking, need, value, etc. – in terms of the relevant percent selecting an item. This is also consistent with what we have seen when comparing selection, yes/no and top 2 box in message filtering (3).

average scores

The rank order of the concepts on each measure was very similar, if not identical, across all measures. That means we would come to the same conclusions with each of the measures.

Sensitivity—measuring differences between messages and attributes

One of the biggest challenges researchers face is being able to differentiate between ideas in terms of which one is the “winner”. We value a measure that discriminates between ideas and that also shows differences across the attributes.

So we measured the degree of difference between concepts and across attributes. In order to level the playing field across the measures, we looked at the differences between the highest and lowest scores, as a percentage of the highest score.

highest and lowest scores

 

Selection showed more sensitivity on both the difference between messages and across attributes. Yes/no also appears to be somewhat more sensitive when comparing between ideas, compared to a top 2 box analysis. We can conclude that selection is more sensitive than the more typical top 2 box analysis of scaled data.

Time to answer

We also measured how long it took respondents to complete idea filtering for the scaled and selection arms of the study. We used the median time to complete for the comparison, because sometimes people pause the completion of a survey part way through and that distorts the average—sometimes quite dramatically. The selection method took less than a minute per idea—twice as fast as the scaled approach.

median time

Conclusion

Our results indicate it is possible to use a selection approach to idea screening that is quick and easy to answer, reliable and sensitive to differences between ideas and leads to the same conclusion as the traditional scaled approach.

Given these results, and in light of how mobile friendly and quick to answer the question is, we recommend using a selection approach to idea filtering.

Methodology

We conducted three studies using sample from our Springboard America market community, which has been proven to be reliable.(4) The samples were representative of the American population and surveys were in English. The sample sizes were: Scaled Arm 505; Selection Arm 503; Yes/No Arm 641.


Questions asked

Selection

Which of the following describe this idea? You can choose as many or as few as you feel describe it.

I’d buy it
I like it
It’s new and different
I need this type of product
I want this type of product
It offers good value
It’s believable
None of the above

Yes/No

Based on the product information you just saw; which statement below best describes how you feel about buying
this product?

Yes
No

I’d buy it
I like it
It’s new and different
I need this type of product
I want this type of product
It offers good value
It’s believable

Scaled

Based on the product information you just saw; which statement below best describes how you feel about buying this product?

Definitely would buy
Probably would buy
Might or might not buy
Probably would not buy
Definitely would not buy

Which statement best describes how much you like or dislike this product?

Like extremely
Like very well
Like quite well
Like somewhat
Like slightly
Not like at all

Considering the price of the product, as given in the information, which statement below best describes how you feel about the value for the money of this product?

Very good value
Fairly good value
Average value
Somewhat poor value
Very poor value

How would you rate this product in terms of being new and different from other products currently available?

Extremely new and different
Very new and different
Somewhat new and different
Slightly new and different
Not at all new and different

How much do you agree or disagree with the following statement about this product?

[ROWS][RANDOMIZE]
I NEED this type of product
I WANT this type of product

[COLUMNS]
Agree strongly
Agree somewhat
Disagree somewhat
Disagree strongly

Which statements listed below best describes how you feel about the believability of the statements made about this product?

Completely believable
Somewhat believable
Neither believable nor unbelievable
Somewhat unbelievable
Completely unbelievable

———————————————————————————————————————-

  1. A Grenville, What Works Better, Scaled or Binary Brand Ratings? Vision Critical University
  2. A Grenville, Changing times demand rethinking old approaches: a case for quick, reliable, easy to answer questions.
  3. A Grenville, Message Filter, Evidence of a Sensitive and Easy Answer Approach to Message and Claims and Testing.
  4. See for example R. Berger and A. Grenville, Wine, Cheese, Scotch and Sample: Know the Source and R. Berger and A. Grenville, Art or Science? The Perils and Possibilities of Survey Sampling in the Evolving Online World.

Idea Filter works efficiently because we have stripped the evaluation down to core measures and streamlined the question-asking process by eliminating cumbersome scales that slow down and often confuse guests. The result is information that is the same as, or more sensitive than, scaled data.

next post thumbnail

Next: