Information Theory: There Are Two Flavors of Uncertainty in Our Lives - Math Helps with Both

In the June (No. 2, 2013) issue of Nautilus magazine, Santa Fe Institute Research Fellow (SFI specializes in complexity science) Simon DeDeo describes how both natural and social uncertainty impact our lives and how mathematics offers a tool to make uncertainty a little less uncertain. 

Here is a brief summary of the article from the SFI website:

DeDeo touches on chance in art and literature, information theory, and the differences between uncertainty arising from the objective material world and that arising from thinking, competing agents. 

"One might imagine a science fiction device—a probability meter—that would measure the differential contribution of nature and humankind to the uncertainty of an outcome," DeDeo writes. "How much uncertainty in the average corporate boardroom is due to nature (e.g., the chances of bad weather delaying a shipment of parts) and how much to the strategic creation of uncertainties by human participants (e.g., the refusal to disclose final production targets to one’s suppliers)? Of course, such a meter could wreak havoc in any real boardroom—and itself be a source of its own readings." 

• Read issue 2 of Nautilus magazine, "Uncertainty" (June 2013)
• Read the inaugural issue of Nautilus, "Human Uniqueness" (May 2013)
• Read an article in The New York Times about Nautilus magazine (May 6, 2013)

Nautilus: Science Connected is a new magazine that examines, from a variety of perspectives, how science is connected to our lives. The articles combine the sciences, culture, and philosophy into a transdisciplinary story "told by the world’s leading thinkers and writers."

This is an excellent find - I am grateful to the folks at SFI for pointing it out in their news blog.

The Coin Toss and the Love Triangle

Information Theory: There are two flavors of uncertainty in our lives, Math helps with both

BY SIMON DEDEO

“I returned, and saw under the sun, that the race is not to the swift, nor the battle to the strong, neither yet bread to the wise, nor yet riches to men of understanding, nor yet favour to men of skill; but time and chance happeneth to them all.” (Ecclesiastes 9:11, King James Bible [Pure Cambridge Authorized Version])

Chance appears to name a single, unitary thing. But its genealogy, its family history, turns out to be a tangled one. One way to understand its branching origins is to turn to literature: We may look, in turn, to two very different novels.

Anton Chigurh, the antagonist of Cormac McCarthy’s novel No Country for Old Men, forces his victims to guess the outcome of a coin toss, taking their life if they guess in error. McCarthy’s villain forces blind chance into his victims’ lives in the most brutal way. That chance is entirely contained, not in Chigurh, but in the toss—in nature itself. This is one source of uncertainty.

To understand the second source, we travel as far as possible from McCarthy’s American Southwest. The first volume of Henry James’s The Wings of the Dove ends with Milly Theale, a wealthy American heiress, visiting the National Gallery in London. To her surprise she sees an acquaintance, Merton Densher, in the company of her best friend, Kate Croy. The plot of the book from this point forward hinges on a single question: will Milly learn what the reader already knows—that Merton and Kate are in love and secretly engaged to be married.

In the sequence, told from Milly’s point of view, we see how Kate—caught sharing an intimate afternoon—acts in such a way as to generate an alternate hypothesis for her friend: that Merton may well be keen on her, but that she, Kate, is not keen on him.

Here are uncertainties not otherwise found in nature: probabilities about probabilities, beliefs about beliefs held by others.

For many of us, the material world of the coin toss may be a byword for chance. But the cognitive world of the love triangle is just as fraught and, at its limits, just as much a form of chance as a tumbling coin. We humans are capable of introducing a degree of uncertainty both dizzying and unavoidable.

Therefore, Ecclesiastes was right twice over. Uncertainties—both natural and human—must be dealt with even by the swift, strong, wise, and able. Our objective is to demonstrate the depths of each kind of uncertainty and to introduce mathematics that can help us come to terms with both.

I. THE COIN TOSS

“What’s the most you ever saw lost on a coin toss?”
(No Country for Old Men, Cormac McCarthy [2005])

Sufficiently many fictional murderers have toyed with their victims that, today, the idea borders on cliché and parody. But Chigurh’s game chills us still, perhaps because of the concentrated form of his message. His impartial, deadly coin toss is a reminder of the dominance of chance in our own lives.

From the accidents we have avoided or been met with, to the relationships we have formed and the institutions we have come to be associated with—each fact about ourselves appears to depend on a series of events, any one of which could have gone another way. To stand on top of a tower of such coincidences and look down gives us a hint of vertigo. Which aspects of life are real, we ask, and which simply luck?

This question is, on the one hand, at the heart of much literature and art; on the other, it is a hardheaded question in the mathematical sciences.

In particular, the branch of mathematics known as information theory concerns itself with how to describe chance and uncertainty. It reconciles the unlikeliness of any particular life with the intuitive sense that the shape of one’s life is not simply a matter of chance.

To see how, let us begin at the beginning, at the base of the tower of coincidences. Suppose Chigurh’s coin toss is fair—equally likely to come out heads or tails. What are the possible strategies his victim should consider? Always guess tails? Heads? Some alternation?

Should the toss be fair, no strategy is better than another. Indeed, strategies are indistinguishable from each other. This is due to the symmetry of the problem: We can swap the labels on each side of Chigurh’s coin at any time and thereby convert any prescription into any other. In McCarthy’s desert landscape, strategy and reason are meaningless. The vertigo of chance is upon us.

This, of course, doesn’t describe the world we normally confront. In our world there is predictability, with a preference between outcomes1 and the possibility of rational choice. Broken symmetries make reason relevant to our lives.

But this doesn’t rescue us from the vertigo of chance. Suppose, for example, Chigurh’s coin is slightly more likely to come up heads. Now, strategies are distinguishable. For a heads-biased coin, the preferred and most rational strategy is always to guess heads.

In the course of your life you need to make many decisions. What if you had to guess the coin toss more than once? Consider tossing the biased coin a thousand times. Since each toss is independent of the one before, the most likely outcome is the repeated occurrence of the most likely outcome of each one separately. And thus, of all the possible histories we could foresee, the strangest sequence of all, an unbroken run of heads:

HHHHH ... H, 1,000 times

is the single most likely. Our intuition is that such a sequence will never, in fact, occur.

And of course we are correct. Fix the bias of the coin at 60 percent heads. The chance of heads on a first toss is (by definition) 6 in 10; of two heads in a row, a little over 1 in 3. The chance of three heads in a row is only a little better than 1 in 5.

The chances of an unbroken run decrease exponentially; the chance of ten heads in a row is less than 1 percent, and a few more doublings suffice to place the chances beyond the astronomical. It is unlikely to see an unbroken run of heads in 80 tosses, even if one completes such a sequence once every second for the 13 billion years the universe has existed.

We’ve established that a string of unbroken heads is extremely unlikely. But any other history is even more so. As time passes, every narrative becomes an extreme rarity. Despite the existence of a most rational choice, the particulars of your life describe a very unlikely path. The biased coin, which signifies the possibility of reason, does not relieve our vertigo after all.

Yet we also know that some things are routine, expected—even, at times, part of our birthright—and others less ordinary. A friendship might depend on having shared a freshman seminar, but is it so unlikely to have made a friend in college?

This intuition has a mathematical grounding in what is called the typical set. The typical set is the mathematics behind our feeling of normalcy. It reconciles reason and chance by linking the nature of the singular event to the properties of the history it belongs to. More than biased probabilities, it challenges the vertigo of pure chance.

Consider the space of all possible histories: an exhaustive list of every sequence of events that might have occurred. Every coin toss, every decision ever made. The typical set bounds a very small region in this space and describes the path that we, as time goes on, are increasingly likely to follow. Given a prescription for the probabilities of individual decisions, the typical set picks a list of histories whose rates of uncertainty accumulation become increasingly close to each other, and match, on the average, the intrinsic rate of the probabilities themselves2.

Let’s return to the freshman seminar. Your college life is a series of chance events whose probabilities are set by a finite list of constraints: your major, your age, and so on. As your college days run on, the set of possible histories you are likely to experience converges with those found in the typical set your constraints define. Each individual history is rich in idiosyncrasy while being drawn from a narrowly circumscribed set of possibilities that is much smaller than the space of all that might happen3.

We are left in a profoundly ambiguous place. The typical set rescues normalcy but also dictates typical lives, common stories: boy meets girl, dog bites man. This wisdom was also known to the author of Ecclesiastes, who wrote that there was “nothing new under the sun.” To the swiftest, the race might go once, or even twice, but on the longest scales of time no streak is left unbroken.

At the same time, our pasts and our futures—even our most likely futures—are, in their details, profoundly unlikely things.

An information theorist may not know the exact world we live in, but she does know that, in the long run, it’s a world in the typical set. And she also knows that she doesn’t know anything else4.

So much for coin tosses and freshman seminar assignments: things external to ourselves, driven by the chances of the physical world5. These material things, however, are not the only—or even the most important—sources of life’s uncertainty. To see that, we turn from Cormac McCarthy’s American Southwest to Henry James’s London.

Read the whole interesting article.