When you toss a fair coin infinitely many times, the frequency of heads will be exactly 1/2 with probability {1} (aka `almost surely’). That’s the strong law of large numbers, and it has a precise mathematical formulation. What this law means is less clear. A sequence of outcomes with different frequency of heads is not logically impossible, so what does it mean that the probability of getting such a sequence is zero ? Is this an assertion about the physical world ? Is there some law of nature that says that the frequency of heads must turn out to be exactly 1/2 ?

The issue of interpretation of probability is puzzling from several aspects. The very fact that mathematical assertions about probability call for interpretation is remarkable: Unlike other creatures that appear in math theorems, such as Hilbert spaces and Lie groups, we don’t view coins and probabilities as exclusively mathematical concepts. When we say that the frequency of heads will be almost surely {1/2}, we feel that we say something about a situation in which a coin is tossed, but it’s hard to formalize what exactly it is that we say.

I don’t know a satisfactory answer to this perplexity, but in this post I want to sketch one approach — the game theoretic foundations of probability, which I only recently became aware of. I learned it from the works of Shafer and Vovk. This approach says that an event {E} has probability zero iff it is possible to design a gambling strategy that will keep your fortune nonnegative regardless of what happens, and make you infinitely rich if {E} occur. Let me exemplify this for the event `the coin will always land on heads’: Suppose that a gambler starts with 1$, and before every coin toss he bets all his money that the outcome will be heads. This gambler will never get into debt — when he loses all his money he stops gambling — and he becomes infinitely rich if the sequence of outcomes is always heads.

Here is the same idea with more latex code (thanks, Luca !): A gambling strategy is a function {g:\{0,1\}^{<\infty}\rightarrow\mathbb{R}}: After observing the finite sequence of outcomes {x_0,x_1,\dots,x_{n-1}\in\{0,1\}}, you bet an amount {g(x_0,\dots,x_{n-1})} that the next outcome will be heads. Here {\{0,1\}^{<\infty}} is the set of finite sequences of Heads ({0}) and Tails ({1}), including the empty sequence. If you start with 1$ and the realized sequence of outcomes of the coin tosses is {\omega=(x_0,x_1,\dots)} then after stage {n} you have

\displaystyle M_n(g,\omega)=1+\sum_{k=0}^n(-1)^{x_k}g(x_0,\dots,x_{k-1})

dollars. The following theorem was first formulated by Jean Ville

Theorem Let {\mu} be the probability measure over {\{0,1\}^\infty} which corresponds to i.i.d tosses of a fair coin. Then the following conditions are equivalent:

  1. {\mu(E)=0}.
  2. There exists a gambling strategy {g:\{0,1\}^{<\infty}\rightarrow \mathbb{R}} such that {M_n(g,\omega)\geq 0} for every {n} and {\omega}, and {\lim_{n\rightarrow\infty}M_n(g,\omega)=\infty} for every {\omega\in E}.

That the second condition implies the first is an easy martingale argument. The other direction, which is also not difficult to prove, is the new part (at least new for readers like me, who studied probability from the measure theoretic approach that governs in math departments). In a way, the game theoretic approach to probability views martingale as a more fundamental notion than probability: Occurrence of a probability zero event is reduced to the possibility of becoming infinitely rich via fair gambling. Note that in the second condition there is no reference to probabilities — it is based on the notion of gambling strategies, which is assumed to be less conceptually problematic.