3 Key Takeaways From The 2017 Advanced Research Techniques (ART) Forum

Advanced Research Techniques

At Maru/Matchbox we are committed to continuously improving the insights we deliver to our clients, and that’s apparent in our attendance at leading market research conferences. We recently attended the Advanced Research Techniques (ART) Forum, organized by the American Marketing Association.

The ART Forum is an important conference in the field of advanced analytics because it brings together academics and research practitioners in a way that does not happen elsewhere. I, with Jack Horne, presented a paper on using Kalman Filtering, with Twitter data, to improve the reliability of tracking results—which I will write about in another article.

In this piece, I summarize three presentations from the ART Forum: one that won best paper, another that won best poster and a third that struck me as particularly promising. We’ll start with the winner of the best paper award.

A first-ever validation of individual reliability in choice models

“Reconciling Stated and Revealed Preferences”, by Hardt, Y Kim, Joo, J Kim and Allenby, is the first study to demonstrate that preferences estimated using choice models from surveys can match preferences derived from modeling what individual people actually buy.

Choice modeling is typically applied to surveys in which people are presented with descriptions of the products featuring multiple attributes. In this case, the research was about frozen pizza. So, the attributes people were exposed to in the survey included brand, the number of people the pizza would feed, the types of toppings, the type of crust, whether the toppings were dense or regular and whether the pizza claimed to be made of “real” cheese or not.

A group of 181 people who are members of a loyalty card program were surveyed and presented with descriptions of various brands of pizza and asked which, if any, they would buy. Two years later, these survey results were then compared to the actual frozen pizza purchases by these same people, as recorded by the loyalty card. The pizzas were then coded to their attributes and the same modeling was done with both sets of data.

The idea behind the modeling is that, by understanding which attributes are present or absent when a choice is made, you can derive the importance of each attribute. With the survey data, the researchers—from Ohio State University, the Korea University Business School and NEOMA Business School in France —were modeling stated preference. With the purchase data they were looking at revealed preference. The primary question the researchers were interested in was: “When do stated and revealed preferences agree?”

Critics of choice modelling contend that the exercise of making these kinds of choices in a survey is too artificial. They feel presenting consumers with lists of attributes does not match with the real world of greater selection, the existence of packaging, varying products to choose from, and the lower likelihood that people will focus on the products attributes in the same way they would a survey. All fair points.

While there has been lots of criticism, many researchers have continued to use choice models. The assumption is that these types of models do indeed provide reliable information about an individual’s preferences and elasticities, and are very helpful when it comes to understanding how to optimize new products not available in the market. But there has been very little proof—beyond market level aggregate data—until now.

This research found “strong agreement” between the survey modeling and the real world data, in terms of the relative importance of the various attributes of the pizza. This provides support for the argument that this kind of modeling works well. The researchers at this conference—many of whom do a lot of choice modeling—welcomed this news and rewarded the authors with the “best paper” award. It certainly is a great first step in an important line of investigation.

Inspired by nature

Peter Kurz won the award for “best poster” with his piece entitled “Computational Intelligence in Product-line Optimization: Simulations and Applications”. Portfolio optimization in discrete choice studies—particularly in complex nested data sets—is problematic, because of the number of possible combinations involved. “Normally you end up with millions of possible combinations of product levels and attributes, therefore the search space is too large for exhaustive product searches,” according to Kurz.

Seeking to find a better solution, Kurz tested 5 algorithms “inspired by nature”. Here is how he described them: “Inspired from Nature: Gradient free optimization algorithms mimic mechanisms observed in nature or use heuristics. Gradient free methods are not necessarily guaranteed to find the true global optimal solution, but they are able to find many good solutions.

  • Genetic Algorithm: Simulation of selection (survival of the fittest), recombination (crossover) and mutation (variation) like in the evolution.
  • Particle Swarm Optimization: Stochastic, population-based computer algorithm that applies to swarm intelligence (simulation of fish or bird-swarms).
  • Ant-Colony Optimization: Motivated by the search for an optimal path in a graph, based on the behavior of ants seeking a path between their colony and a source of food.
  • Simulated Annealing: Inspiration come from annealing in metallurgy, a technique involving heating and controlled cooling of a material to increase the size of its crystals and reduce their defects.
  • Multiverse Optimizer: Main inspirations of this algorithm are based on three concepts in cosmology: white hole, black hole, and wormhole.”

In a test using simulated data, Simulated Annealing performed best, followed by Multiverse Optimization and the Genetic Algorithm. In a test using data from a complex conjoint study the results were different. The Multiverse Optimizer was a clear winner. It found the highest market shares for all runs and all interactions. It took, however, 14 to 15 times longer to run than the fastest algorithm.

These findings suggest a number of “inspired by nature” algorithms offer useful alternatives to the commonly used Genetic Algorithm. The difference in results between the simulated data and the conjoint example indicate that the individual algorithms might be more or less effective, depending upon the type of data being analyzed.

This paper provides useful evidence that multiple kinds of “inspired by nature” algorithms have the potential to effectively address the problem of product-line optimization.

Faster estimation opens the door to wider use

Iterative Multilevel Empirical Bayes has the potential to make choice modeling become something done on the fly, according to a paper by Kevin Lattery. Called “Iterative Multilevel Empirical Bayes (IMEB): Flexible and Robust Solution for Large Scale Conjoint”, this paper is the latest in a series in which Lattery has chronicled his quest to make Empirical Bayes a viable and valuable tool for choice modeling.

Hierarchical Bayes is more commonly used than Empirical Bayes because it has been shown to produce a better model fit and has higher hit rates for prediction. However, because Hierarchical Bayes can require 20,000 to 100,000 iterations before hitting an optimal solution, it is time consuming. Models can take—depending upon the sample size—hours, if not days, to run.

This paper demonstrated how it is possible to use a Fisher Information Hessian matrix to turn Empirical Bayes into an iterative process, allowing the model to improve itself in relatively few iterations (20 to 30). This results in dramatically reducing the estimation process from hours to a few seconds, making it possible to complete model estimation for individual respondents on the fly. This opens the door to a variety of potential uses. It could, for example, be used in adaptive surveys—where your choices will determine what you are exposed to next.

The need for fewer iterations also makes it considerably less time consuming to analyze large data sets. In one test, with a sample of 1,200 people, the Hierarchical Bayes model took over 74 hours to run, while the Empirical Bayes took less than 10th of that time: 6.5 hrs.

While there are still issues to be solved regarding this method, such as incorporating covariates, it’s great to see the potential in this technique.

Also notable were papers on: predicting bundle preference using configuration data; monetizing ratings data for product research; perceptual choice experiments; a Bayesian price sensitivity meter and predictive text analytics.

It was encouraging to see so many researchers working towards improving the power of analytics. We look forward to harnessing new techniques to deliver great insights for you.