The Skeptics Society & Skeptic magazine


Evidence:
“It Worked for My Aunt Tillie” Is Not Enough

Bee pollinating a flower at the National Zoo

ABOUT THE IMAGE above: A bee pollinating an Echinacea flower, by Moxfyre (Own work) [CC BY-SA 3.0 or GFDL], via Wikimedia Commons

Evidence means different things to different people. Even quacks and their victims claim to have evidence that their treatments work. Sometimes that evidence consists only of testimonials from satisfied customers or from personal experience. “I tried X and I got better.” “I know Y works because it cured my Aunt Tillie’s arthritis.”

I had a friend who used all kinds of questionable treatments including homeopathy. I asked her how she decided what to try. She said if a friend told her something had worked for him, and if it didn’t seem dangerous, she would try it. That was all the evidence she needed. She didn’t care about scientific evidence because she said, “Science doesn’t know everything.” Comedian Dara Ó Briain had the perfect answer to that: “Science knows it doesn’t know everything; otherwise, it’d stop. But just because science doesn’t know everything doesn’t mean you can fill in the gaps with whatever fairy tale most appeals to you.” When Oprah Winfrey told Jenny McCarthy that experts said there was no scientific evidence that vaccines caused autism, Jenny retorted, “My science is named Evan, and he’s at home. That’s my science.”

There is a huge disconnect between what science-based medicine calls evidence and what alternative medicine and the general public call evidence. They are using the same word, but speaking a different language, making communication next to impossible.

First, there is no such thing as “alternative medicine.” There is only medicine that has been tested and proven to work and medicine that hasn’t. If a treatment currently considered to be alternative were adequately tested and proven to work, it would be incorporated into mainstream medical practice and could no longer be considered “alternative.” It would become just “medicine.” So-called “alternative” medicine can be defined as medicine that isn’t supported by good enough evidence to earn it a place in mainstream medicine.

“Alternative medicine,” along with “complementary and alternative medicine” (CAM) and “integrative medicine,” is not a meaningful scientific term, but a marketing term created to lend respectability to things that we used to call by less respectable names like quackery, folk medicine, and fringe medicine. It puts everything outside of science-based medicine under a single umbrella, including things that might work but haven’t been adequately tested, things that have been tested and proven not to work but that are still being used (like applied kinesiology—a bogus muscle testing procedure used by many chiropractors), and fanciful things that couldn’t possibly work, like homeopathy. It fosters the misleading impression that all these things might be equally valid.

There is no such thing as “alternative medicine.” There is only medicine that has been tested and proven to work and medicine that hasn’t.

Science-based medicine has one rigorous standard of evidence, the kind of evidence government agencies require before they allow a pharmaceutical to be sold. CAM has a double standard. They gladly accept a lower standard of evidence for treatments they believe in. However, I suspect they would reject a pharmaceutical if it were approved for marketing on the kind of evidence they accept for CAM.

When science-based medicine evaluates the evidence for alternative medicine, here is what it concludes:

  • Acupuncture is just a theatrical placebo.
  • Homeopathy not only doesn’t work but couldn’t possibly work.
  • Chiropractic is essentially physical therapy contaminated with bogus diagnostic and treatment methods.
  • Energy medicine is fantasy medicine dressed up with sciencey-sounding words like “quantum” and “frequencies.”
  • Naturopathy claims to stress prevention and to consider the whole patient, but that’s just what every good doctor does. It offers “natural” treatments that have not been tested or that have been proven not to work, like homeopathy. What naturopaths do that is good is not special and what they do that is special is not good.
  • Herbal medicine is plausible: after all, half of our prescription drugs came from plants. But every herbal remedy must be tested individually to determine if it is effective and safe; and statistically the great majority of promising remedies fail testing.

So while science says the evidence is lacking, CAM says there is plenty of evidence:

  • They cite preclinical studies in animals and test tubes, but scientists recognize that even the most promising preclinical studies can be misleading and must be confirmed in humans.
  • They cite case reports, but we know that case reports are only useful as indicators of what science should evaluate with controlled studies.
  • Sometimes they can cite clinical studies in humans, but those studies are often flawed. They may be poorly designed or lack a control group, they may be contradicted by other studies, they may be based on such extraordinary claims that we would have to have extraordinary evidence to accept them, and they don’t build on each other to create a coherent body of evidence.
  • They cite doctors who say, “In my experience, X is effective.” Mark Crislip, one of my colleagues on the Science-Based Medicine blog, says “in my experience” are the three most dangerous words in medicine, because personal experience is so very compelling and so often leads to false beliefs.
  • They rely heavily on testimonials and anecdotal evidence. Scientists know that the plural of anecdote is not data; no matter how many testimonials you accumulate, they can’t ever prove that the treatment works. Think of all the testimonials over the centuries for balancing the humors with bloodletting.

CAM believers find their evidence very convincing, and it is hard to explain why we don’t and how their kind of evidence has been shown over and over to lead people to false conclusions.

Scientists know that the plural of anecdote is not data; no matter how many testimonials you accumulate, they can’t ever prove that the treatment works.

You can’t blame people for accepting that kind of evidence. They are only doing what comes naturally. Science doesn’t come naturally to humans. Evolution equipped us with thinking processes that facilitated survival on the African savannah. Our ancestors had only two sources of information: their own experience and the experiences of other people. If someone said “Don’t eat those berries; they are poisonous,” there was no way to verify the information; but listening to others was likely to help keep you alive. Humans became very adept at pattern recognition. If a pattern of shadows in the bushes looked sort of like a lion, it was safer to assume it was a lion and run away. The cost of over-interpreting patterns was small, while missing the pattern of a real lion could be fatal. Our ancestors learned to make quick decisions: if it was really a lion, they needed to run away now. And emotions served as motivators: fear made them run away faster.

Those thinking processes helped our ancestors survive in a prehistoric environment, but they have become a handicap in the modern world. Today we have more sources of information, but our minds still work the old way. We prefer stories to studies, anecdotes to analyses. We see patterns where none exist. We jump to false conclusions based on insufficient evidence. Emotions trump facts. If your neighbor had a bad experience with a Toyota, you’re likely to remember his story and not buy a Toyota even if Consumer Reports says it’s the most reliable brand. That isn’t logical, but humans are not Vulcans. When we act illogically, we’re just doing what evolution has equipped us to do. It takes a lot of education and discipline to overcome our natural tendencies, and not everyone can do it.

Ray Hyman is a psychologist and one of the founders of modern skepticism. When I asked him why some people become skeptics and others don’t, he said he thinks skeptics are mutants: something has evolved in our brains to facilitate critical thinking. Is it nature or nurture? Or both? Maybe life experiences also incline some of us to think more critically.

So what kind of evidence should persuade us? It’s tricky, because, as John Ioannidis has shown us, many published research findings are wrong. Early pilot studies are overturned by larger, better-designed studies. Studies are influenced by researcher bias. Some are even fraudulent. With over 23,000 scientific journals and 700,000 papers published every year, it’s easy to find a study to support almost any belief.

In this context, there is a hierarchy of evidence:

  • Basic science.
  • Test tube studies (in vitro).
  • Animal studies (in vivo).
  • Case reports—of a single patient.
  • Case series—reporting on a number of patients.
  • Case control studies (example: comparing people with and without lung cancer to see if there are more smokers in the group with lung cancer).
  • Cohort studies (example: following people who smoke and who don’t smoke over a period of time to see which group develops more lung cancers).
  • Epidemiologic studies (example: studying whether people in countries with more smokers develop more lung cancers. These studies can show correlations, but they can’t determine causation. Countries with more smokers might have other confounding factors that predispose to lung cancer).
  • Randomized controlled trials (RCTs).
  • Systematic reviews and meta-analyses that evaluate all the published evidence pro and con.
Skeptic magazine issue 20.3 (cover)

This article appeared in Skeptic magazine 20.3 (2015).

The best studies are large, randomized, placebo-controlled, and double- blinded; but that isn’t always necessary or even possible. We don’t need an RCT to know that parachutes save lives. It’s unethical to knowingly endanger a control group by denying them effective treatment. (Anyway, who would volunteer to jump without a parachute?) We didn’t need RCTs to figure out that smoking causes lung cancer because there is an overwhelming body of corroborating evidence from several different kinds of research.

Evidence matters. Science works. It’s a collaborative, self-correcting enterprise. It never relies on a single study. When a finding is true, it will be corroborated by other studies, and a consensus will eventually build based on a cohesive body of evidence. False findings are eventually overturned and discarded. We can trust the scientific method; we can’t trust anecdotes. Sorry, Aunt Tillie! END

About the Author

Dr. Harriet Hall, MD, the SkepDoc, is a retired family physician and Air Force Colonel living in Puyallup, WA. She writes about alternative medicine, pseudoscience, quackery, and critical thinking. She is a contributing editor to both Skeptic and Skeptical Inquirer, an advisor to the Quackwatch website, and an editor of Sciencebasedmedicine.org, where she writes an article every Tuesday. She is author of Women Aren’t Supposed to Fly: The Memoirs of a Female Flight Surgeon. Her website is SkepDoc.info.

55 Comments
Newest
Oldest
Inline Feedbacks
View all comments

This site uses Akismet to reduce spam. Learn how Akismet processes your comment data. Comments are closed 45 days after an article is published.

Donate

Get eSkeptic

Be in the know!

Subscribe to eSkeptic: our free email newsletter and get great podcasts, videos, reviews and articles from Skeptic magazine, announcements, and more in your inbox twice a week. It’s free. We never share your address. Unsubscribe any time.

Sign me up!

Retrospective

Skeptic cover art by Pat Linse

Art of the Skeptic

In celebration of Skeptic magazine’s 100th issue, we present sage graphic art advice for skeptical groups and a gallery of art reflecting more than 47 years of skeptical activism from Skeptic’s long time Art Director, Pat Linse

Detecting Baloney

Baloney Detection Kit Sandwich (Infographic) by Deanna and Skylar (High Tech High Media Arts, San Diego, CA)

The Baloney Detection Kit Sandwich (Infographic)

For a class project, a pair of 11th grade physics students created the infographic shown below, inspired by Michael Shermer’s Baloney Detection Kit: a 16-page booklet designed to hone your critical thinking skills.

FREE PDF Download

Wisdom of Harriet Hall

Top 10 Things to Know About Alternative Medicine

Harriet Hall M.D. discusses: alternative versus conventional medicine, flu fear mongering, chiropractic, vaccines and autism, placebo effect, diet, homeopathy, acupuncture, “natural remedies,” and detoxification.

FREE Video Series

Science Based Medicine vs. Alternative Medicine

Science Based Medicine vs. Alternative Medicine

Understanding the difference could save your life! In this superb 10-part video lecture series, Harriet Hall M.D., contrasts science-based medicine with so-called “complementary and alternative” methods.

FREE PDF Download

Top 10 Myths of Terrorism

Is Terrorism an Existential Threat?

This free booklet reveals 10 myths that explain why terrorism is not a threat to our way of life or our survival.

FREE PDF Download

The Top 10 Weirdest Things

The Top Ten Strangest Beliefs

Michael Shermer has compiled a list of the top 10 strangest beliefs that he has encountered in his quarter century as a professional skeptic.

FREE PDF Download

Reality Check: How Science Deniers Threaten Our Future (paperback cover)

Who believes them? Why? How can you tell if they’re true?

What is a conspiracy theory, why do people believe in them, and can you tell the difference between a true conspiracy and a false one?

FREE PDF Download

The Science Behind Why People See Ghosts

The Science Behind Why People See Ghosts

Mind altering experiences are one of the foundations of widespread belief in the paranormal. But as skeptics are well aware, accepting them as reality can be dangerous…

FREE PDF Download

Top 10 Myths About Evolution

Top 10 Myths About Evolution (and how we know it really happened)

If humans came from apes, why aren’t apes evolving into humans? Find out in this pamphlet!

FREE PDF Download

Learn to be a Psychic in 10 Easy Lessons

Learn to do Psychic “Cold Reading” in 10
Easy Lessons

Psychic readings and fortunetelling are an ancient art — a combination of acting and psychological manipulation.

FREE PDF Download

The Yeti or Abominable Snowman

5 Cryptid Cards

Download and print 5 Cryptid Cards created by Junior Skeptic Editor Daniel Loxton. Creatures include: The Yeti, Griffin, Sasquatch/Bigfoot, Loch Ness Monster, and the Cadborosaurus.

Copyright © 1992–2021. All rights reserved. | P.O. Box 338 | Altadena, CA, 91001 | 1-626-794-3119. The Skeptics Society is a non-profit, member-supported 501(c)(3) organization (ID # 95-4550781) whose mission is to promote science & reason. As an Amazon Associate, we earn from qualifying purchases. Privacy Policy.