Testing, testing . . . .


In his book Bad Science, columnist and psychiatrist Ben Goldacre presents the following simple problem. (I have modified it a bit, but the basic idea is the same.)

I have a deck of 100 index cards. On one side of each card is a letter; on the other is a number between 1 and 10 (inclusive). That information you can take as true. I then deal out four cards as follows.

cardsNow I ask you to test the following hypothesis as well as you can by turning over just two cards.

Hypothesis: Every card in the deck that has a vowel on the letter side has an even number on the number side.

Question: Which two cards do you flip?

I leave you to mull that over before you go on.

Most people start by flipping the A card. If there  is an even number on the other side, that  supports the hypothesis. If it is an odd number, the hypothesis is shown to be false. In fact, there is a 2 on the other side, so we have support (but not proof!) of the hypothesis.

Hardly anyone flips the B card. Since B is not a vowel, the parity (even or odd) of the other side will tell you nothing about the hypothesis, which says nothing about the other side of non-vowel cards. In fact, there is a 4 on the other side.

What about the 2 card? If you find a vowel on the other side, you will have gathered support (but not proof!) for the hypothesis. But if it’s a non-vowel, you have no evidence for or against the hypothesis. Still, a lot of people turn over the 2 card. And find an F.

On to the 3 card. If the other side is a non-vowel, then the situation is similar to flipping the B card. This result neither supports nor denies the hypothesis. But if the other side is a vowel, then the hypothesis goes down the drain. You have a vowel on one side and an odd number on the other. Definite proof that the hypothesis is untrue.

Do you see the point here? If the hypothesis true, you will have to go through the whole deck to confirm it. But it takes one exception (a non-vowel on the last card) to prove that the hypothesis is untrue.

In short, you should turn over the A and the 3 cards because they are the ones that will show the hypothesis to be untrue. Don’t bother turning the 2 card. Incidentally, here’s what you see when you turn over all the cards.

cards2

This conclusion leaves a lot of people baffled. For them, testing a hypothesis amounts to searching for evidence in favour of it. Looking for exceptions strikes them as a kind of betrayal. Deliberately avoiding evidence against the hypothesis is so common it has a name — confirmation bias.

Advertisements

About aharmlessdrudge

Way back during the late Bronze age -- actually it was the 1950s -- all of us in high school had to take a vocational test to determine our interests and, supposedly, our future careers. I cannot remember the outcome, but I do recall one question that gave me pause. "If you were to win a Nobel prize, would it be in literature or in physics?" I hesitated over the question: although I enjoyed mathematics and science more than English class, I did have a couple of unfinished (and very bad) novels hidden away at home. I cannot remember what I chose back then, but the dilemma followed me to university, where I switched from mathematics to English and -- after a five-year stint in journalism -- back to mathematics. I recently retired as a professor of statistics. Retirement. What a good chance to revive my literary ambitions. I have finished a novel -- more about that in good time -- and a rubble of drafts of articles about mathematics and statistics is taking up space on my hard disk.
This entry was posted in Crank science, Science and Medicine, Skepticism and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s