Noise in Decision-Making: Why Experts Disagree

This isn’t the usual article about decision-making. You know the type: “you need better data so invest in Big Data” or “artificial intelligence is the answer to everything.” No, it’s about how experienced people can reach very different judgements from the same information… and how those differences can go unnoticed.

All because of something called Noise.

Many business decisions depend on professional judgement: weighing incomplete evidence, risk and competing needs. In Noise, Daniel Kahneman, Olivier Sibony and Cass Sunstein show why we should pay attention not only to biased judgement, but also to unwanted differences between judgements that ought to be consistent. As Kahneman puts it, “Wherever there is judgment, there is noise, and there is more of it than you think.”

Noise is unwanted variability in judgements that should be reasonably consistent.

It is easy to miss when people make their assessments separately and nobody compares the answers.

For consequential decisions, it is worth checking how much judgements differ, why they differ and whether the differences are justified.

What is decision noise?

In Noise (2021), Kahneman, Sibony and Sunstein explore how this unwanted variation can affect professional judgement.

We’ve all heard about bias: a systematic tendency for judgements to skew in one direction. Noise is different. It is the spread of answers when comparable cases should receive reasonably consistent assessments. The illustration below contrasts the two.

In a 2016 Harvard Business Review article, Kahneman and colleagues reported research in which the same software developers estimated the same task on two different days. Their time estimates differed by 71% on average. Even the same person's estimate could change substantially between occasions.

Kahneman also described an insurance-company noise audit. Underwriters given the same cases produced premium estimates that differed by about 55%, compared with the roughly 10% variation executives expected. The gap exposed a consistency problem the company had not recognised.

Six things to know about noise and what to do about it

1. Inconsistent judgements can have a real cost

If similar cases get very different assessments, the consequences can reach beyond the decision itself: inconsistent prices, missed opportunities or resources committed on shaky assumptions. When Kahneman asked the insurance company's executives to estimate the cost of setting premiums too high or too low, they put it in the billions of dollars.

2. Disagreement can stay invisible

At a global engineering company we worked with, different engineers scored a particular project for complexity. Half gave it the lowest possible score while the other half gave it the highest possible score.

If the team sees only an agreed score, this disagreement could disappear from view. The next step is to explore whether the engineers used different definitions, evidence or assumptions, or interpreted the same information differently.

3. Try a simple consistency check

Give several experts the same case, information and scoring instructions. Ask for independent judgements, then compare the answers. A big gap tells you where to investigate, not which person is right.

• Different interpretations? Are people scoring the same thing? If some mean technical difficulty and others mean organisational disruption, clarify the criterion and assess again.
• Different evidence? Does one expert know something the case information left out, such as a dependency or risk? Check the information before trying to agree a score.
• Different priorities? If people genuinely value different outcomes, make that disagreement explicit for the people responsible for the choice. It is not simply a scoring error.

If a wide gap remains once the question, evidence and priorities are clear, you may be looking at noise itself: capable people reaching inconsistent judgements from the same case.

4. A single confident answer can hide uncertainty

When a judgement is presented as one definitive score, it can look more settled than the underlying evidence warrants. Comparing independent estimates makes that variation visible while there is still time to challenge the inputs.

5. You can reduce noise

Averaging independent judgements of the same case can reduce noise, because some individual variations cancel out. As Kahneman explains, it does not remove a bias everyone shares. And it only makes sense to combine answers that are genuinely comparable.

Use the consistency check above first: clear up different readings of the question, bring missing evidence into the open and let decision-makers address real differences in priorities. An average can help summarise comparable judgements; it cannot settle a disagreement about what matters.

6. Get independent judgements before the group discussion

When several people are assessing the same question, ask them to record their initial judgements before discussing them together. That reduces the chance that an early or influential opinion will anchor everyone else. Then compare the results, understand the differences and make the decision collaboratively.

Flag-waving: where TransparentChoice fits

Normally in these posts, I don’t talk too much about TransparentChoice, but this is different.

TransparentChoice provides collaborative decision-making software that can collect independent assessments of the same initiative and let people compare responses before discussing them. This makes differences in judgement visible so the team can explore them.

TransparentChoice also uses the Analytic Hierarchy Process (AHP) to structure criteria and stakeholder judgements. That gives a team a clearer basis for comparing and challenging assessments. For the method, see our Guide to Analytic Hierarchy Process.