The problem with being data-driven

And why you should strive for an evidence-based approach instead

I’ve always disliked the expression “data-driven organization”. To me, companies that want to move away from making decisions based on opinions should strive to develop an evidence-based approach rather than become more data-driven. Here’s why.

Case study: When a data-driven initiative backfires

A few years ago I was conducting some interviews with sales managers at car dealerships. The goal was to understand how sales teams used data to make decisions at the local level, as input to help shape an analytics product. A sales manager at a dealership of a popular luxury car manufacturer offered the perfect example of a data-driven initiative that produced the reverse of the expected effect.

A data analysis made at the corporate headquarters had shown a high correlation between a seller having six calls with a prospect and that person buying a luxury car. A new policy was instituted across all dealerships: "Sellers must make six calls to each of their prospects.” Only after some time had passed under the new rule with negative impact on sales, the top executive in charge agreed to remove the policy.

One can imagine someone at headquarters looking at the patterns arising from sales data across dealerships and getting excited: "Oooh! 95% of our wins had a seller calling the buyer six times. Meanwhile, among our losses, the seller only called 2-3 times. There is a big opportunity for us to increase our win rate just by doubling the number of calls made to each prospect."

This is a typical case of a confounding situation, where the relationship observed among two variables (number of calls and sales success/failure) is actually provoked by a third variable that isn't in the picture.

When a positive correlation between number of calls and sales outcome was identified, before celebrating the finding and rushing to create a new policy, the decision-makers should have looked into two possible explanations:

Possibility #1: Making six calls to a prospect cause them to buy a luxury car.

Possibility #2: An unmeasurable third variable influences both the supposed cause (six calls) and the supposed effect (successful sale of a luxury car).

Had corporate executives talked to a few sellers, they’d quickly realize that the positive correlation between number of calls and successful sale resulted from a prospect with high intent to buy engaging in conversation with a seller and welcoming multiple calls that clarified options, confirmed price estimates, etc. before closing the deal. Insisting that sellers increased the number of calls to random prospects who didn’t belong to the same segment risked turning off other potential buyers, such as individuals who were considering the brand along with others and simply expecting a quick response when they reached out with questions via social media.

You may be thinking that this was an obvious, avoidable blunder. Unfortunately, this story is far too common, especially in organizations where leadership is told that success hinges on becoming more “data-driven” without any effort to ensure the decision-makers are comfortable with analytical thinking.

Many data-driven companies end up data-rich but insight-poor

As seen with the example of the luxury car manufacturer, investing a ton of resources on acquiring and crunching data doesn’t mean a company will be able to adequately answer performance questions or improve its decision-making process. In fact, in this example a “data-driven” decision hurt the bottom line by alienating some prospects who didn’t like the aggressive approach of sellers initiating unwanted calls.

Measuring things and gathering data is a means to an end. Taken together, collected data points can show patterns that result in new opportunities and better solutions to existing problems—but only if leaders understand the attributes of an evidence-based analytical decision process.

The journey from data-driven to evidence-based

The most intelligent decisions are not data-driven, but rather the result of data-informed judgments made by a multidisciplinary team making a conscientious, explicit, and judicious use of current best evidence.

To help their organizations evolve from data-driven to evidence-based, leaders need to learn to differentiate data used to pursue operational excellence from data needed to develop breakthrough solutions and allow rigorous statistical thinking to be combined with observation, experience, and creativity in support of high-quality decisions and actions.

Leaders who adopt an evidence-based approach accept the fact that data may look actionable even when it is incomplete or biased. While there will always be uncertainty and ambiguity in the data used to make decisions, evidence-based leaders spend time examining their data and their thinking, involve cross-functional teams in the analysis, and ask tough questions to help their organizations to more confidently make decisions amid uncertainty.

Photo by Jamie Street on Unsplash