Wednesday, January 27, 2010

Do We Have an Issue with Overconfidence?

I somehow had missed the current crisis of overconfidence. Granted, overconfidence is what has made homo sapiens the dominant species on the planet. On the other hand, it likely contributed to the housing crisis, the hedge fund mess, and various other economic and cultural disasters.

The authors of the research, see below, actually propose that overconfidence is what got us here today, in a good way. As a biologically rewarded trait, it has been passed down from generation to generation, right up to the current day - where it may not be as valuable an adaptation.

It is this last possibility that a writer at Fortune has chosen to highlight, calling it a "crisis." Maybe. But I am skeptical at best. The financial crisis is less about overconfidence than it is deregulation and just plain old greed.

Anyway, Dominic D. P. Johnson and James H. Fowler published a paper on The Evolution of Overconfidence, which seems to have generated some interest.
ABSTRACT
Confidence is an essential ingredient of success in a wide range of domains including job performance, mental health, sports, business, and combat. Many authors have suggested that overconfidence--defined here as believing you are better than you are in reality--is advantageous because it serves to increase ambition, resolve, morale, persistence, and/or the bluffing of opponents. However, too much overconfidence can cause arrogance, market bubbles, financial collapses, policy failures, disasters, and wars, so it remains a puzzle how such a false belief could evolve or remain stable in a population of competing accurate beliefs. Here, we present an evolutionary model that shows overconfidence actually maximizes individual fitness and populations will tend to become overconfident, as long as the resources at stake during conflicts exceed twice the cost of competition. This is because overconfident individuals make more challenges when there is uncertainty about the strength of opponents (and thus the outcome of conflicts), while less confident individuals shy away from many conflicts they would win. Where the value of a prize is at least twice the cost of trying, overconfidence is the best strategy. The model suggests that the conditions under which humans would have evolved to have a "rational" unbiased view of their own capabilities are exceedingly rare, and it helps to explain why resource-rich environments can paradoxically create more conflict. Moreover, the fact that overconfident populations are evolutionarily stable may be one reason why overconfidence persists today in politics, business, and finance, even if it causes occasional disasters.
You can read the whole article as a free PDF.

Fortune ran an article based on this study, in which David Ewing Duncan used the research to explain the financial crisis.

A crisis of overconfidence

Are humans hard-wired for overconfidence - and does that help explain the financial crisis?

By David Ewing Duncan, contributor
tight_rope.ju.03.jpg

NEW YORK (Fortune) -- Thousands of years ago, our ancestors fanned out from Africa, convinced that something better lay over the next hill. They brazenly faced down a dangerous world of beasts much larger and fiercer than they were, and their boldness as hunters probably contributed to some species -- saber-toothed cats and the mammoth, for instance -- going extinct.

While there may appear to be little similarity between those prehistoric times and today, it turns out that the same audacity that made us effective predators also may have driven the relentless advance of Homo sapiens from caves to condos and from primitive haggling to hedge funds.

Arguably, it's our overconfidence that's kept us going, and now a new paper titled "The Evolution of Overconfidence" from researchers in Scotland and San Diego suggests that this distinctly human trait may be vital to our success.

But this predilection to overestimate one's abilities, adversaries, and outcomes also has a dark side, since not everyone can be the best merely by thinking they are. Take drivers: Over 90% believe they are above average at maneuvering an automobile. Or college professors: 94% of them claim they are above average teachers.

While some may certainly be assessing themselves accurately, that simply can't be true of all of them. But their overconfidence can still be an asset, boosting resolve, ambition, morale, and tenacity, even for the average driver or professor.

Yet because overconfidence by definition makes it difficult to accurately evaluate risk, it has also contributed to disasters that range from getting eaten by long-fanged felines 15,000 years ago to global warming, the Vietnam War, and the catastrophic response to Katrina.

In the recent financial collapse, we had an epidemic of overconfidence in just about every quarter (realtors, homebuyers, investors, bankers, lawmakers, and the list goes on). In the lead-up to the crash a year ago, the warning signs were clear as debt piled-up, housing prices and sales began to slide, and market upswings became detached from the real economy.

Still, as late as July 2008, Treasury Secretary Henry Paulson was saying: "Our banking system is a safe and a sound one."

The human proclivity to overconfidence has led experimental psychologists to puzzle over how a trait that can be so destructive remains so potent in our gene pool.

One explanation comes from the authors of "The Evolution of Overconfidence," evolutionary biologist Dominic Johnson of the University of Edinburgh and political scientist James Fowler of the University of California at San Diego. Using a mathematical model that compares the overconfident with the more tempered, they conclude that the audacious usually reap more rewards than those who rationally assess risk.

Johnson and Fowler calculate that when the potential upside is at least twice that of the cost to compete, overconfidence is the best strategy. "This is because overconfident individuals make more challenges when there is uncertainty about the strength of opponents (and thus the outcome of conflicts)," write Johnson and Fowler, "while less confident individuals shy away from many conflicts they would win."

If true, this contradicts the prevailing economic and social theory of the last 30 or 40 years that individuals and societies tend to make rational choices, balancing costs and benefits in a reasonable manner -- whether they are buying a new car or going to war.

The Johnson-Fowler scenario may also tell us why the rational choice theory seems to have failed so spectacularly in the past year or two. Seen through their model, financial traders were taking ever more fantastic risks -- usually with other people's money -- apparently believing they would win big. Like drivers, most assumed they would be not only above average, but way above average.

But as we've learned, there's an outer limit that seems to occur when overconfidence becomes too extreme and the ratio of costs to rewards becomes dangerous or potentially suicidal. Ancient Greeks called this brand of overconfidence "hubris" -- an excessive pride, confidence, and ambition that Greek myths and stories constantly warn against for both mortals and gods. (There were no hedge fund traders back then.)

But there's no formula to predict the moment when overconfidence becomes hubris, and optimization reels into disaster. Had there been, our distant ancestors might have avoided killing off every last one of those saber-toothed cats -- and in today's turbulent (and self-inflicted) economic times, we might have some sense of just how brash we need to be to repair the damage done by our own brashness.
Interestingly, it is Duncan, and not Dominic D. P. Johnson and James H. Fowler, the authors of the study, who showed up on Tech Nation to talk about the study, also back in December.

David Ewing Duncan

Evolution of Overconfidence
12 minutes, 5.9mb, recorded 2009-11-17
Dr. Moira Gunn speaks with author and columnist, David Ewing Duncan and gets in the scientific scoop on what he calls, the evolution of overconfidence.

No comments: