Let's Hear What the Rock Has to Say

I haven’t been able to get this post from Scott Alexander out of my head, both because it is interesting and because it goes in a direction I did not expect.

The post is series of whimsical scenarios where a human is employed to use judgement, but in fact a simple heuristic outperforms human judgement. The heuristic is good enough a getting the probabilities right that it would be more accurate to replace human judgment with a rock with the heuristic inscribed on it. This snippet will give you the feel:

The Interviewer

He assesses candidates for a big company. He chooses whoever went to the best college and has the longest experience.

Other interviewers will sometimes choose a diamond in the rough, or take a chance on someone with a less-polished resume who seems like a good culture fit. Not him. Anyone who went to an Ivy is better than anyone who went to State U is better than anyone who went to community college. Anyone with ten years’ experience is better than anyone with five is better than anyone with one. You can tell him about all your cool extracurricular projects and out-of-the-box accomplishments, and he will remain unswayed.

It cannot be denied that the employees he hires are very good. But when he dies, the coroner discovers that his head has a rock saying “HIRE PEOPLE FROM GOOD COLLEGES WITH LOTS OF EXPERIENCE” where his brain should be.

Alexander then worries about experts using simple heuristics to mimic expertise without true ability.

But the heuristics are so hard to beat that the experts themselves might be tempted to secretly rely on them, while publicly pretending to use more subtle forms of expertise. “My statistical model, accounting for chaos theory, barometric pressure, and the price of tea in China, says there won’t be a hurricane tomorrow. Rejoice!”

I believe that the examples are all purposefully chosen such that the heuristic is a variation on “rare events rarely happen” but the frequency of the event is the inverse of its importance. So you don’t really want a stupid rock running things, even though 99% of the time you do, because it only give p(x) not the expected value of x.

I am surprised at the direction of this worry. My experience with humans is that they don’t give nearly enough thought to the heuristic and that they way overestimate their own judgement.

We have a non-whimsical example of a case where a pretty simple heuristic often beats human judgement pretty soundly, and the human charges you extra for the pleasure of losing you money: active fund managers. These experts aren’t even in the business of predicting rare events, and still manage to underperform the heuristic of “just put all your money in an S&P 500 Index fund”. Active fund managers would be putting money back into their clients’ pockets if the would just let the rock decide. But they don’t! And people still give them plenty of money, too. Go down to you local Edward Jones and they will happily suggest active funds which take more fees to make investors less money.

Maybe people just can’t bring themselves to believe that simple heuristics can work, or maybe some experts love clout more than they love accuracy. Predicting boring stuff like “the weak form of the efficient market hypothesis is true” will make you money, but slowly and no one will remember those calls. If you want clout, you’ll have to make calls people will remember, for good or for ill. For clout purposes, you really aren’t graded on calls no one is paying attention to.

My suspicion is that the existence of Simple Heuristics that are Almost Always Right is a blessing. Things are so much easier evaluate when you have a simple default to evaluate against. Sports analysts have replacement level players, financial analysts use the risk free rate of return. It’s not enough to say that one expert is better than another, it’s preferable to establish that either of them are better than a dumb, easy to follow rule.

I find that in everyday decision making, a dumb heuristic to be a good cross test against motivated reasoning. I’ve been involved with many business decisions people offer up predictions that would only come to pass if consumers weren’t sensitive to price changes. And sometimes they aren’t, but not nearly as frequently as these predictions are offered. If the prediction hinges on Demand curves not sloping downward, it’s a good time to go back and ask if this a thing you think is true, or just a thing you want to be true. “Demand curves slope downward” isn’t always right, but it’s right often enough that it should only be overruled when there’s evidence enough to discard it.

So that’s where my intuition is different from Alexander’s. I take it that engagement with these kinds of heuristics is a signal both the appropriate amount of subject-matter depth and humility. Disengagement is a signal of either lack of depth or lack of humility, or both.

Rocks are good! Use the rocks, don’t be afraid of other people using the rocks to save themselves some work.