Legacy customer experience and voice of customer tracking systems are showing their age, to the point where large research budgets are yielding fewer insights as the program ages. In this article, we explore how to ensure your tracker stays flexible and meaningful to capture relevant and timely customer insights.
Our Research on Research (RoR) study
As part of our continuing efforts to stay true to getting meaningful customer-centric insights, Maru/Matchbox recently executed research on research (RoR) in the customer experience space to understand the best way to capture and evaluate experiences as customers see and live through them.
We evaluated Uber rideshare experiences and kept the study in context to respondents’ overall customer experiences with the brand over the past several months.
Aside from evaluating overall satisfaction, brand advocacy and brand favorability, we also evaluated Uber experiences on a number of other components that typically get factored in when consumers evaluate a rideshare experience, including the app, their drivers, price and discounts and the vehicles used, among others.
CX Program Design Challenges
One big challenge with designing a comprehensive good customer experience program is knowing what dimensions of experience to evaluate. Typically, we would recommend:
- Leveraging past research and known customer pain and delight points.
- Conducting qualitative research with customers to understand which aspects of the relationship need to be evaluated and how they can best be measured to ensure they are actionable.
However, we do not always have the luxury of being able to do such groundwork due to limited client budgets or lack of legacy data to reference. In such cases, CX study design is based on the collective domain knowledge of clients and our experienced researchers (in addition to social chatter) to understand what aspects should be measured, quantified and analyzed on an ongoing basis to keep a pulse on customer relationship performance.
It also becomes even more important to find a way to ensure the aspects that are being measured are still relevant to customers over time. One way to do that is to have a way in which to funnel customer feedback verbatim back into study metric design.
Given that our Uber RoR study also sought open-ended contextual customer feedback, it served us well to understand how CX studies can stay nimble enough to change and fine tune what is being measured based on what customers are saying and feeling about the brand.
How We Approached Our Unstructured Client Feedback Data
Using Maru/Matchbox’s text analytics tool, which allows us to theme and categorize open-ended responses using logical and conditional rules, we came up with 18 different themes from customer comments. One way to incorporate such unstructured data into survey design would be to simply include the themes being mentioned the most.
Going by the frequency of mentions alone would suggest we should include themes that were represented at 5% or higher to ensure the survey doesn’t become too troublesome for the respondent. But, in our study, doing this would have ended up:
A. Overlapping a fair bit with metrics that were already covered in the initial survey design.
B. Adding elements that were talked about frequently but not necessarily having a big impact on the overall experience (e.g. “Gets me to my destination/Meets my needs” may be mentioned often but may not be a big driver of overall satisfaction, because it’s really a basic expectation of the outcome of a service you are paying for).
A better way to assess whether an open-ended theme should be added as a metric in the survey would be by looking at the relationship that the themes may have with your dependent measure, in this case “Overall Satisfaction”.
Running a correlation between our themes and overall satisfaction quickly confirmed the lack of a strong relationship for most measures. There was, however, one that stood out: ‘Corporate Governance’; the following table shows a clear inverse relationship between the incidence of this theme and overall satisfaction.
Some examples of comments that fell into this theme:The ‘Corporate Governance’ theme largely represented customers’ views on Uber corporate policies and practices, especially those related to employees/contractors and surge pricing.
- “Uber’s rates often go up during peak times or bad weather which can make them more costly than a regular taxi. I’m also not happy with Uber’s corporate employment practices because I feel it takes advantage of its drivers by classifying them as contractors.”
- “I think they control the market and don’t take care of the contractors that work for them. Everyone was okay with them charging less than taxis but now many of those contractors are out of a job and Uber hasn’t helped them because there aren’t any health benefits, vacation, etc.”
- “I fundamentally disagree with surge pricing.”
Running a Shapley value regression driver analysis further confirmed that out of all themes, ‘Corporate Governance’ was the one that had the highest relative importance in driving satisfaction. While we wouldn’t necessarily recommend using text theme-based driver results directly, these results sufficiently suggest that this is a metric worth adding, and tracking, to assess Uber performance on an ongoing basis.
Additionally, it is a metric that could realistically be easily missed in the original design, being less of an experience-related metric and more of a brand-related one, but the importance of its inclusion was able to be identified by analyzing unstructured data in a quantified way.
There are two key takeaways from this analysis that will be worth factoring into your CX tracking programs:
- Don’t be complacent with experiential attributes you already have in your CX survey. Customer experiences and the ways that customers evaluate you are always changing based on evolving needs and circumstances. In this case, Uber policies toward their drivers seem to be relevant to customers’ evaluations of their experiences, especially in light of the current socio-economic environment as well as joblessness caused by the COVID-19 pandemic.
- The usage of unstructured data in a more quantified way, to see what relationships hidden themes might have with the overall experience, is very useful in zeroing in on what evaluating metrics may be important to add to a living, breathing assessment of the customer relationship.
As we, at Maru/Matchbox, develop additional tools to analyze unstructured data, including our recently released, machine learning-based topic modeling functionalities, we anticipate this process of identifying and incorporating new themes into our bespoke CX surveys to become even more efficient and insightful over time. We feel this is imperative to ensure that our clients’ CX programs are aligned to their customers’ changing evaluation criteria where it comes to how they feel about our clients.
Contact us to learn how we can apply our CX expertise to help you get the most out of your CX tracking systems.