Lowering uncertainty with limited data
With a little less defeatism and more creativity, all of us can make better decisions
American executive Jim Barksdale once said words that serve as an excellent incentive for people to find data to reduce uncertainty any time they’re making a consequential decision:
After all, who wants important decisions being made based on the judgement of one person or very few individuals operating under the highest levels of uncertainty?
In reality, though, many organizations mistakenly treat the problem of evidence-gathering as a binary problem. They either decide to go after every piece of information available before making an important decision, or, if facing time or resource constraints, delegate even the most strategic decisions to the highest-paid person’s opinion (HiPPO).
Often the optimal solution lies between these two extremes. Adopting a sophisticated and comprehensive method of data collection isn’t necessarily going to improve decision-making (see example in When Imperfect Data is Your Friend). On the other hand, that doesn’t mean we shouldn’t try to find a simpler measurement approach to help us draw solid conclusions that reduce uncertainty and improve the outcome of important decisions.
To reduce uncertainty, a little creativity can go a long way
Imagine that you’re in a band that uses heavy special effect equipment in its shows. Not all stages will support the heavy weight, so your contract includes numerous pages describing the requirements for the venue to prevent gear from falling over the audience or the members of the band. There isn’t enough time, before each show, to line-check the entire production. How do you reduce uncertainty when deciding whether it’s safe to go on stage?
The famous ‘no brown-colored M&Ms’ rule
NPR Music tells the story:
It's one of those rock 'n' roll legends that turns out to be true: In the 1980s, the party-rock superstars in Van Halen demanded, via a clause embedded in their tour rider, that no brown-colored M&Ms be allowed backstage at their concerts.
While the demand was initially seen as purely rock star diva act, the real point was safety. Van Halen’s touring rig was heavy, so their touring contract imposed several conditions that each gig venue had to meet to avoid injury and equipment damage if the venue's floor didn’t support the weight.
The clause about M&Ms, placed well after the weight ones, was a test of dilligence.
No bowl of M&Ms sans brown ones backstage was a sign that the contract had not been read and the place couldn’t be considered safe.
This is an excellent example of how a simple, creative heuristic can help mitigate risks. Could a venue still fail to meet all requirements even when the ‘no brown-colored M&Ms’ rule was satisfied? Sure, the test didn’t offer a 100% confidence level. Nonetheless, it reduced the uncertainty when making the decision to proceed or not with the show compared to merely using “gut feeling”.
A creative way to find the right restaurant location
Restaurant owners know that having a good location is critical. Of course, what constitutes an attractive location will vary depending on who the restaurant wants to appeal to, whether the menu is cheaply priced or upscale, etc.
The book Restaurant Owners Uncorked offers a great example of how a restaurant owner reduced uncertainty with minimal data. Phil Roberts, a legend in the restaurant business who launched over a dozen successful restaurants including The Occeanaire Seafood Room, was asked by a potential investor what was his strategy for opening more seafood places.
“Well you show me a Morton’s that’s doing about $7,000,000. and you show me a Ruth’s Chris that’s doing about $ 7,000,000. I’ll put The Occeanaire down right between the two of them and create the ultimate surf and turf.” The investor asked, “That’s your location strategy?” and Roberts answered, “You’re damn right it is. That’s all I need.”
How to tell when a little bit of data is all you need
No brown M&Ms and find a surf ‘n turf location are simple heuristics used to save effort while substantially reducing the uncertainty associated with an important decision. But you may be wondering, When is it OK to use a “shortcut” or “rule of thumb” like that to make an important decision? When it would be irresponsible not to go after ALL the data before making a critical decision?
The work of Gerd Gigerenzer and Wolfgang Gaissmaier offers some good clues. Their research confirms that using simple heuristics that ignore part of the information can lead to even more accurate judgments than weighting and adding all information, in particular under the following circumstances:
Heuristics are likely to do well in an environment with moderate to high uncertainty and moderate to high redundancy (that is, the different data series available are correlated with each other).
An example they cite from previous research successfully illustrates their point.
For retailers, it’s valuable to know which customers will be back or have abandoned their business for good.
Turns out that while there’s a lot of uncertainty over if and when a customer will buy again, the time since last purchase tends to be closely correlated with every other available metric of past customer behavior. While the amount of data that retailers are able to collect grows and predictive models keep growing, we can expect much of it to be highly correlated (e.g., the longer the time since last purchase, the lower the number of times the customer is likely to have opened a promotional email, used a coupon, or logged in to their account over the past X months).
Hiatus heuristic: If a customer has not purchased within a certain number of months (the hiatus), the customer is classified as inactive; otherwise, the customer is classified as active.
Studies quoted by the authors demonstrated empirically a less-is-more effect: a complex model to classify a customer as inactive made more errors despite performing extensive estimations and computations using significant more information than the hiatus heuristic.
How to improve the quality of decisions when we only have limited access to data
Granted, not all circumstances will fit the “sweet spot” for the heuristics approach discussed above. There are times when more data would be ideal, but the data either doesn’t exist or isn’t in a structured, easily analyzed data set.
That doesn’t mean we can’t use creativity and analytical thinking to improve upon the HiPPO scenario when we’re facing an important decision.
Acumen, a global nonprofit that invests in early-stage companies whose products and services enable the poor to transform their lives, asks the question, If we could measure and understand the differences between the ways women and men experience poverty, could that enable us to make better decisions as business leaders, program managers, and investors striving for gender equity?
Gathering data to determine the differences on how men and women experience poverty can be costly and time-consuming, but Acumen used its Lean Data methodology to support data collection and analysis in a quick, cheap, robust, and actionable way.
As Douglas Hubard, author of How to Measure Anything wrote,
You have far more data than you think. Assume the information you need to answer the question is somewhere within your reach and if you just took the time to think about it, you might find it. (…) The things you care about measuring are also things that tend to leave tracks, if you’re resourceful enough to find them.