Simple Statistical Rules Outperform Intuitive 'Clinical Judgment'

Clinical vs. Statistical Prediction: A Theoretical Analysis and a Review of the Evidence
, Paul Meehl

The Meehl pattern
Why are experts inferior to algorithms?
- Simple statistical rules outperform intuitive 'clinical judgment.'

1.
It's because experts, when making predictions, try to be smart, think creatively, and consider complex combinations of multiple features.
Complexity may work for unusual cases but often undermines validity.

Some studies have shown that decision-makers, even when given a score suggested by a formula and asked to predict, performed worse than the prediction formula itself!

In a famous thought experiment, Meehl described a formula for predicting whether someone will go to a movie tonight, and pointed out that if you receive the information that the person broke their leg today, it is appropriate to disregard the formula. From this comes the broken-leg rule. The point of this rule is that a broken leg is decisive, but extremely unusual.

2.
Another reason expert judgment is inferior is that, when summarizing complex information into a judgment, humans are chronically inconsistent. When asked to make the same judgment on the same information twice, we often give very different answers. The degree of this inconsistency is a hugely important issue.

Among veteran radiologists who read chest X-rays as 'normal or abnormal,' 20% of the time they read the same image differently when shown it in different contexts.


Meehl's research presents a striking conclusion.
If you want to maximize predictive accuracy, you should leave the final decision to the formula. This is especially true in low-validity environments.

For example, when predicting the future value of unaged wine, the experts who evaluate quality usually have access to information that, on the whole, is more likely to make the situation worse rather than better. That's because they can taste the wine. And even if they understand well how weather affects wine quality, they can't keep up with the consistency of a formula.