The Law of Large Numbers says that “the average of the results of a large number of trials should be close to the expected value”. This is common knowledge, but most people are not familiar with the corollary that “the average of the results of a small number of trials may be quite far from the expected value”.

I recently read “Thinking, Fast and Slow” by Daniel Kahneman. It’s a very interesting book about the biases we have and the errors we automatically make when thinking. The book consists of example after example of how our heuristics aren’t as good as we think they are. One particularly striking example involves the incidence of kidney cancer in counties in the US (this article seems to be the basis the chapter in the book).

Consider this: A study of the incidence of kidney cancer in the 3,141 counties of the United States reveals a remarkable pattern. The counties in which the incidence of kidney cancer is lowest are mostly rural, sparsely populated, and located in traditionally Republican states in the Midwest, the South, and the West. Now, what do you make of this information?

Now consider the counties in which the incidence of kidney cancer is highest. These ailing counties tend to be mostly rural, sparsely populated, and located in traditionally Republican states in the Midwest, the South, and the West.

So, the extreme (cancer-free and cancer-ridden) cases seem to be those where the sample population was smallest. Once you think about it for a bit, this isn’t really surprising. We can do a simple test to see that this is true.

Suppose we roll a dice; if it comes out \(6\), you have cancer. We roll \(N\) dice for a sample of \(N\) people. We’re interested in how many of the \(N\) people in each sample have cancer; more specifically, we’re interested in how many of the samples are “extreme” (i.e. where everybody in the sample is either cancer-free or has cancer). When \(N = 2\), we have two people in each sample, and we see that extreme samples account for \(72\%\) of all samples (we’re considering all the possible samples – all the possible ways of rolling \(N\) dice).

When \(N = 6\), we have six people in each sample, and we see that extreme samples account for \(33\%\) of all samples. It’s important to note that only the sample size has changed – the probability that any person has cancer is unchanged.

Just by increasing the sample size from \(2\) to \(6\), we see that the proportion of extreme samples has gone down from \(72\%\) to \(33\%\). This trend will keep on going as we increase the sample size.

To sum it all up, extreme cases are more likely to turn up if the sample sizes are small.