The Value and Danger of Anecdotal Experiences

Why anecdotes shouldn't be trusted. (xkcd #842: Mark)

I would like to share with you an anecdote.

For a long time, I kept my cell phone in my left pants pocket. After some time, I began noticing that the muscle just underneath it would occasionally twitch.

Feeling curious, I moved my phone to my right pocket. After a while, I noticed that the twitches in the left muscle weren’t happening anymore. And just recently (about 6 months later) I noticed some twitches occurring under my right pocket.

Ah-ha! you might think. Long-term exposure to cell phone radiation has some effect on muscles!

Well… no. It’s a curious correlation at best, but there are a number of problems with making any sort of conclusion from this single experience:

Am I giving you a true account of my experience? I could easily have made all this up. Maybe I’m looking for attention, or money, or have some other interest driving me. Or I could just be a pathological liar.

Confirmation bias. Even if I am being completely honest with you about my memory, my memory could well be flawed. People have a tendency to notice events that agree with their preconceived notions and ignore those which do not. This can easily happen subconsciously. Twitches which occurred under my pocket with the cell phone would be more likely to rise to conscious awareness than those occurring elsewhere, including on the other side without the cell phone. (More on confirmation bias)

Recall bias. In addition to confirmation bias, people are more likely to remember and put weight on an anomalous event than to the surrounding normalcy. In my case, I am noticing the twitch happening, but not the normal, non-twitching state that preceded and followed it. (More on recall bias)

Confounding factors and randomness. Even if all the rest was accurate, was it really the cell phone radiation? What about its wireless network signal, or the pressure of the phone against my leg, or the heat emitted by the phone/retained in the muscle? Also, even though I don’t think I’ve had twitches in those particular places in other times, it is entirely possible that the (N=~2) association is a coincidence. There is absolutely no way to tell.

Anecdotal reports are not evidence. For these reasons and more, anecdotal reports are not evidence. They may allow us to generate hypotheses, but until one or more experiments are done which minimize the effects of biases and can control for unrelated variables, there’s no way to know if my anecdote is based in reality or is no more than an active imagination making an inappropriate extrapolation. And certainly does not provide a sound basis to start fearing cell phones.

If someone wants to use their own anecdotal experience for themselves, they are free to do that. But there are even fewer grounds for any other person to accept someone else's anecdote.

Anecdotes are powerful and can be misused. Personal stories are very powerful in influencing the way we think, in part because we can form emotional attachments to the subjects of the story. Politicians use them to drive a point home, and snake oil salesmen rely on them to push sketchy and unverified treatments (see my post on homeopathic medicine, a system which is almost entirely based on anecdotes). And after all, almost everyone has heard the false story of someone being trapped by a seatbelt, despite the evidence that such situations are much rarer than accidents where restraint would save their life. But the ability to imagine yourself in the described dangerous situation is hard to ignore despite the rational basis provided by a statistically-powerful study (this is part of negativity bias).

We don’t need to discard all anecdotal accounts, but we do need to look at those offered to us as evidence with a skeptical eye, and ensure that any anecdotes we offer to others are truly representative of some backing of evidence.