Whether managing teams, measuring performance, evaluating opportunities, or deciding where to expand, identifying and tracking the right metrics is an essential part of an Experience Management (XM) program.

But metrics don’t always communicate the information that they are meant to represent. As a result, organizations regularly make mistakes interpreting and using metrics. Bias in our X-data metrics can have serious ramifications for business. It triggers poor investment strategies and exacerbates a myriad of negative behaviors related to cultural, racial, and economic bias.

There are two critical reasons why metrics can become untrustworthy. The first is rather obvious: not picking the right metrics. But the second reason is more dangerous, and likely more prevalent: bias in the collection of experience data (X-data) which goes unidentified and uncontrolled.

Know the Types of X-Data Bias

The first step in detecting and neutralizing bias in your X-data is knowing what kinds of bias to look for. The two most common classes of bias are environmental bias and measurement bias.

Environmental Bias

Environmental bias occurs when the natural environment which we are measuring has characteristics that create skewed results. Many of these characteristics are beyond the control of the frontline managers and employees we often seek to measure in our metrics ecosystem. Environmental bias can unnaturally suppress the scores of teams based on their operating conditions, but it can also unnaturally boost the performance of others. 

Some common examples of environmental characteristics which affect X-data:

• Population Density. Retail stores that sit in highly urban areas tend to have higher foot traffic, more crowded peak times, and higher sales volumes. Lower satisfaction – driven by negative experiences like longer lines or products being temporarily out of stock – is more likely to occur in these environments, even if employees do their best to mitigate issues.
• Experience Format. Simple bank branch characteristics like different floor plan design and type (e.g. how branches are laid out, “digital-only” branches) and store features (e.g. services offered, whether there is an ATM) can have a meaningful impact on a bank’s satisfaction score.
• Degree of Difficulty. Some experiences come with higher expectations than others. “VIP Client” operations, for example, are higher stakes with greater expectations. Even if those experiences are materially better than standard interactions or competitors’ services, higher expectations result in greater frequency of customers being “merely satisfied” or dissatisfied.
• Operational Wildcards. Unpredictable delivery timelines can affect re-stocking capacity in convenience stores, creating the appearance of “product out of stock” even if inventory indicates otherwise. Even real estate choices (e.g. number of parking spaces, proximity to difficult intersections, proximity to public transportation hubs) can affect length of lines, number of employee illnesses, and even average basket contents.

Measurement Bias

Measurement bias occurs when how we measure and represent experiences is skewed by the means of measurement itself. Many of these biases are well-known and are the more thought-of forms of bias in our X-data. These are addressable by the XM Center of Excellence within high-functioning organizations.

Some common examples of measurement bias include sample bias, response bias, cultural scale bias, and modal bias.

• Sample bias. This type of bias results in data that are not representative of the target population. For example, customers with very good or poor experiences may be more likely to give feedback while average experiences are under-represented. 
• Response bias. The way we ask questions in surveys can influence how people respond and, in turn, result in biased data. One common example is acquiescence bias where respondents are naturally inclined to agree with a statement. For example, some people are more likely to answer ‘yes’ in response to questions like ‘the agent was helpful in resolving my issue.’
• Cultural scale bias. People from different cultures respond to the same scale differently, even when controlling for the quality of the experience. For example, American consumers tend to rate satisfaction higher than those of Western Europe, while Japanese consumers give lower grades than Western countries.
• Modal bias. The form factor of X-data collection creates a natural bias in how consumers respond. For example, phone-based surveys produce more socially desirable responses, and order bias can be more common on mobile phone surveys where scrolling is required.

Three Recommendations to Overcome X-Data Bias

Bias in X-data most commonly comes from variables that XM programs routinely control for through weighting schemas (age, gender, income, region, business size, business type, etc.). To control for other types of bias, organizations must take the next step to determine whether there is inherent bias in the X-data, assess its extent, and neutralize it. Here are three strategies for overcoming bias.

Recommendation #1: Detect Bias with Statistics

Using data and insights to manage a business is challenging because the data we read are often interpreted through the lens of our own history, experience, and, therefore, bias. While some people may believe that bias can be easily identified with a “golden gut,” the single most effective way of detecting whether your KPIs are being affected is to use analytics.

Use statistical techniques such as regression and ANOVA to identify which O-data variables (e.g., day-part/week-part or the type of telephony equipment agents use) are having a statistically significant impact on the X-data you track.

This type of analysis might reveal that call center agents who receive low overall satisfaction scores are not necessarily ineffective. Rather, it may be that the shifts they work (such as weekend shifts) are materially influencing the types of calls they receive as well as the mood of the customer calling in.

Recommendation #2: Eliminate Environmental Bias

Most bias in data doesn’t have ill intent or evil origins. Instead, the issue arises when some normal aspect of operating a business goes unaccounted for in the process of goal-setting and benchmarking. We need to eliminate bias that results in some teams not getting enough credit – as well as others receiving credit too easily – especially where compensation is linked.

There are two common tactics to eliminate environmental bias: differential standards and relative standards.

• Use differential standards to compare similar groups. The simplest method of defeating environmental bias is to analyze and use data in like-for-like conditions only. For example, if whether or not a convenience store offers food for sale influences KPIs, only analyze stores that sell food against similar-format stores. This method is best when comparisons within groups are feasible.  
• Use relative standards when separate comparisons aren’t feasible. When it is necessary to analyze data across groups, statistically generate an “expected score” (based on the operating conditions) that controls for bias. Calculate the delta between the “expected” and “actual performance” to create a relative standard that portrays results fairly. 

Recommendation #3: Mitigate Measurement Bias

Measurement bias is predominantly knowable and reducible by applying research methods best practices.

Mitigate measurement bias with activities in two categories:

• Catch up to existing best practices. Upgrade best practices for survey invitations, survey question design, special modules, analysis, ticketing systems, and more to improve the reliability and validity of everyday metrics used to guide organizational change. Some best practices, like the use of relative metrics, naturally target and nullify different types of bias.
• Actively create new best practices. Improve on methodological best practices with persistent A/B/n testing. These tests can focus on iterative improvements to methods (such as improving the scale used in a question) or paradigm changes to methods (using video question types instead of scales). 

The bottom line: As in all facets of life: eliminate bias to have a better, fairer, and more productive system.

Luke Williams is an XM Catalyst with the Qualtrics XM Institute

Moira Dorsey, XMP, is an XM Catalyst with the Qualtrics XM Institute