Tuesday, November 1, 2016

Joseph Berkson on p-values

I'm about to finish Jordan Ellenberg's "How Not To Be Wrong: The Power of Mathematical Thinking". Perhaps, I'll write a slightly longer review later but for now, let me point out that my favorite section of the book has been its discussion of p-values. To demonstrate the problems of over-reliance on p-values, Ellenberg recounts the following:
But here’s the bad news: the reductio ad unlikely, unlike its Aristotelian ancestor, is not logically sound in general. It leads us into its own absurdities. Joseph Berkson, the longtime head of the medical statistics division at the Mayo Clinic, who cultivated (and loudly broadcast) a vigorous skepticism about methodology he thought shaky, offered a famous example demonstrating the pitfalls of the method. Suppose you have a group of fifty experimental subjects, who you hypothesize (H) are human beings. You observe (O) that one of them is an albino. Now, albinism is extremely rare, affecting no more than one in twenty thousand people. So given that H is correct, the chance you’d find an albino among your fifty subjects is quite small, less than 1 in 400,* or 0.0025. So the p-value, the probability of observing O given H, is much lower than .05.
We are inexorably led to conclude, with a high degree of statistical confidence, that H is incorrect: the subjects in the sample are not human beings.

No comments:

Post a Comment