Skeptic » eSkeptic » March 19, 2008

The Skeptics Society & Skeptic magazine

Skepticality: The Official Podcast of Skeptic Magazine
woman drinking water

Toxic Water?

Last week, a widely circulated Associated Press story reported that “a vast array of pharmaceuticals — including antibiotics, anti-convulsants, mood stabilizers and sex hormones — have been found in the drinking water supplies of at least 41 million Americans.”

Derek & Swoopy look beyond the headline hype in their interview with Dr. Shane Snyder, who published his first study on “Screening of Drinking Water for Possible Endocrine Disrupting Compounds” over ten years ago. As a doctor of Environmental Toxicology and Zoology, Dr. Snyder has devoted most of his career to studying the evidence of chemical, environmental and pharmaceutical compounds in the water supply — and what harm, if any, it poses to the organisms that utilize it.

SUBSCRIBE to Skepticality
within iTunes

DOWNLOAD Episode #73 (29MB MP3)

SUBSCRIBE to the Skeptic RSS feed

In this week’s eSkeptic, the SkepDoc herself, Harriet Hall, M.D, reviews R. Barker Bausell’s book entitled Snake Oil Science: The Truth About Complementary and Alternative Medicine (Oxford University Press, 2007 ISBN 9780195313680).

detail of book cover

Snake Oil Science: The Truth About Complementary & Alternative Medicine (detail of cover)

Snake Oil Science

by Harriet Hall, M.D.

I could condense this review into three words: “read this book!”

The term “complementary and alternative medicine” (CAM) is relatively new, but the treatments it encompasses are not. Before we had science, all we had to rely on was testimonials and beliefs. And even today, for most people who believe CAM works, belief is enough. But at some level, the public has now recognized that science matters and people are looking for evidence to support those beliefs. Advocates claim that recent research validates CAM therapies. Does it really? Does the evidence show that any CAM therapy actually works better than placebos? R. Barker Bausell asks that question, does a compellingly thorough investigation, and comes up with a resounding “NO” for an answer.

Bausell is the ideal person to ask such a question. He is a research methodologist: he designs and analyzes research studies for a living. Not only that: he was intimately involved with acupuncture research for the National Center for Complementary and Alternative Medicine (NCCAM). So when he talks about what can go wrong in research and why much of the research on CAM is suspect, he is well worth listening to.

He describes his acupuncture research in great detail. It involved patients with pain from dental surgery. Before designing the experiments, he searched the literature and found an article that reviewed 16 previous trials of acupuncture for dental pain and concluded that it was probably effective. But on the Jadad scale, a simple 5-point measure of quality, none of those 16 studies scored higher than 3 (which is considered barely adequate) and 5 of them incredibly scored zero. Bausell’s group set out to resolve the question with research of much higher quality. For instance, a low dropout rate is one measure of quality; they only had 3 subjects drop out during the course of the study, and those 3 were people the researchers sent home because of a snow storm!

They compared “true” acupuncture to the most credible “sham” acupuncture they could devise. There was no difference in outcome: both were equally effective in relieving pain. When they looked more closely at their data, they found some surprises. The placebo control was not perfect, and some subjects had been able to guess which group they were in. Knowing you really got acupuncture should have increased the placebo response, and knowing you didn’t should have decreased it — yet even so, there was no difference between the groups. So the results were even more negative than they appeared. Even more fascinating, patients who thought they got real acupuncture reported much more pain relief than those who thought they got the sham, regardless of which they actually got!

Bausell points out that penicillin cures pneumonia even if you’re in a coma, but alternative medicine only seems to work when you are awake. You have to know (or think) you’re being treated. And penicillin works by well-understood scientific principles, while much of alternative medicine is based on “entire physiologic systems or physical forces that the average high school science teacher already knew didn’t exist.” If any alternative treatment clearly worked as well as penicillin, prior plausibility wouldn’t matter: science would adopt it and worry about how it worked later. Under the circumstances, prior plausibility is an important consideration.

He tells his mother-in-law’s story. She had knee pain from osteoarthritis with fluctuating symptoms. Every time the pain increased, she would try something new she had read about in Prevention magazine and every time it would seem to work as the pain naturally decreased again. And eventually it would seem to stop working as the pain naturally increased again. She would phone every couple of months to tell him about the wonderful new treatment she had discovered. She was not ignorant or stupid, but she underestimated the power of the placebo and didn’t realize how the natural fluctuations of her pain led her to false conclusions.

She had fallen for the most common human error: the post hoc ergo propter hoc fallacy. The fact that pain relief follows treatment doesn’t necessarily mean that the treatment caused the pain relief. This is only one of the many impediments to correct thinking that plague our fallible human brains. Bausell describes some of those other impediments. He shows how patients, doctors, and researchers are all equally likely to fool themselves, and why the most rigorous science is needed to keep us from reaching false conclusions.

book cover

Bausell’s thorough discussion of the placebo phenomenon is illuminating and invaluable. He covers the history of research on placebos and tells some fascinating anecdotes. He argues that placebo response is not just imagination. It is a learned phenomenon, a conditioned response. You respond to a placebo pill because you have previous experience of being helped by pills. Morphine injections in dogs cause a side effect of salivation: after a while, you can inject water and they will respond with salivation. Physiologic effects from placebo are always smaller than with the real thing, but apparently they do occur. The evidence for objective physiologic effects may not be entirely convincing, but it is certain that pain and other subjective symptoms respond to placebos. And there is even research suggesting a mechanism: the release of endogenous opioids, pain-relieving chemicals produced by our own brains. If you counteract those chemicals with a narcotic antagonist like Narcan, you can block the placebo response.

He shows that the act of taking a pill may really relieve pain, but that the contents of the pill may be irrelevant. Research shows a hierarchy of placebo response: injections work better than capsules and capsules work better than tablets. The color and size of the pill and the frequency of dosing all make a difference. And intriguingly, patients who have responded to a placebo have distortions of memory: they remember the pain relief as greater than it actually was!

Bausell points out that

…just because someone with a PhD or an MD performs a clinical trial doesn’t mean that the trial possesses any credibility whatsoever. In fact, the vast majority of these efforts are worse than worthless because they produce misleading results.

The book includes valuable lessons on how to tell credible research from the other kind. Even the most experienced researchers will find food for thought here, and for the layman it will be a revelation.

Research is full of pitfalls. Negative studies tend not to get published (the file drawer effect). Research done by believers and pharmaceutical companies tends to be more positive than research done by others. Studies from non-English speaking countries are notoriously unreliable for various reasons — 98% of the acupuncture studies from Asia are positive, compared to 30% from Canada, Australia, and New Zealand. The researcher may delegate the actual research to others, who may make undetected mistakes or deliberately skew results to please their boss. Double blind studies may not be truly blind: subjects may have been able to guess which group they were in. Subjects who are not responding may drop out. People who believe in homeopathy are more likely to volunteer for homeopathy studies. Researchers may put a positive spin on their findings or reach conclusions that are not justified by the data. Even if the research is impeccable, we arbitrarily use p=.05 as the measure of statistical significance, and this means there is a 5% probability that the results will appear falsely positive just by chance.

There are more pitfalls, and Bausell covers them all. When you come right down to it, no experiment is beyond criticism, and most published research is wrong. So how can we decide which studies are credible? We now have published guidelines such as the 22 item Consolidated Standards of Reporting Trials (CONSORT) checklist to assess the quality of randomized controlled trials, but Bausell offers some simpler criteria that can rule out the worst offenders:

  • Subjects are randomly assigned to a CAM therapy or a credible placebo
  • At least 50 subjects per group
  • Less than 25% dropout rate
  • Publication in a high-quality, prestigious, peer-reviewed journal

Using this simple 4-item checklist, he reviewed all the CAM studies published in The New England Journal of Medicine and The Journal of the American Medical Association from 2000–2007. 14 met the criteria, and all were negative. When he expanded his search to include the Annals of Internal Medicine and Archives of Internal Medicine, he ended up with 22 studies, only one of which was positive (exactly what you would expect from the 5% rule if none of them worked).

Since different studies have conflicting results, we now use meta-analyses or systematic reviews to try to reach a reliable conclusion. In fact, we even have systematic reviews of systematic reviews! After explaining why systematic reviews are subject to several pitfalls of their own, Bausell goes on to examine the high-quality systematic reviews from the Cochrane collection. Cochrane’s independent reviewers take the quality of studies into account and try to evaluate all the published evidence without bias. Of 98 CAM reviews in the Cochrane database, 21 were positive. When he subtracted those that lacked confirmation by studies in English-speaking countries, those with suspect controls, and those that were subsequently trumped by more definitive high-quality studies, the percentage of positive studies dropped to that familiar 5%.

A highly touted non-Cochrane review of homeopathy concluded that the clinical effects of homeopathy were not just due to placebo. But strangely they also concluded that there was insufficient evidence to show that any single homeopathic treatment was clearly effective in any one clinical condition. A re-analysis of the studies they had reviewed showed that when only the highest quality studies were considered, the alleged positive effect for homeopathy disappeared.

What all this amounts to is that advocates can point to plenty of “snake oil” science that apparently supports various CAM treatments; but when examined critically, the entire body of evidence is compatible with the hypothesis that no CAM method works any better than placebo. True believers will never give up their favorite treatment because of negative evidence; they will always want to try one more study in the hope that it will vindicate their belief. They see science as a method they can take advantage of to convince others that their treatment works. They don’t see it as a method of finding out whether their treatment works. Bausell says

CAM therapists simply do not value (and most, in my experience, do not understand) the scientific process.

He doesn’t aim to dissuade anyone from using CAM. He just doesn’t want anyone to choose it for the wrong reasons, to be fooled into thinking there is credible evidence where there isn’t. He emphasizes that CAM nourishes hope, and its placebos work, if only for symptoms that would eventually resolve on their own anyway. The comfort CAM brings can be valuable, as long as it is not used in place of effective treatments for serious conditions — and most of the time it isn’t, despite the occasional horror story of a patient who refuses effective cancer treatment and dies using a useless remedy. He even has some advice on how to choose a placebo therapy that works for you. He encourages you to believe because otherwise it won’t work.

Bausell writes in an entertaining, accessible style: the book reads almost like a detective story. It should appeal equally to the general reader and to the scientist. The book isn’t perfect, but its minor flaws don’t detract from its message. It is destined to be a classic on the order of Robert Park’s Voodoo Science and Michael Shermer’s Why People Believe Weird Things. If you want to understand how medical research works, if you want to know what can lead patients and scientists to false conclusions, if you have ever used complementary or alternative medicine or have wondered why others do, if you value evidence over belief, if you care about the truth, read this book.

Dr. Michio Kaku

Dr. Michio Kaku will be speaking on THURSDAY, March 27, 2008 at 7 pm
(NOTE the special date and time for this lecture).

the next lecture in our Spring season…

Physics of the Impossible

with Dr. Michio Kaku

SPECIAL EVENT DATE: Thursday, March 27, 2008 at 7 pm
Baxter Lecture Hall, Caltech

One hundred years ago, scientists would have said that lasers, televisions, and the atomic bomb were beyond the realm of physical possibility. In his new book, Physics of the Impossible, the renowned physicist Michio Kaku explores to what extent technologies and devices deemed equally impossible today might become commonplace in the future.… READ MORE about this lecture >

READ MORE about other upcoming lectures >

Important ticket information

Tickets are first come first served at the door. Sorry, no advance ticket sales. Seating is limited. $8 Skeptics Society members & Caltech/JPL Community; $10 General Public. website banner

Every week, we’ll be adding new content to and we’ll announce those additions here. You can also stay up-to-date by subscribing to the RSS feed.

new this week on



  1. Mark Flamand says:

    The author of this article leaves a lot to be desired. This is my 14th year for practicing Reiki. I have personally witnessed 2 people coming out of comas (in an intensive care unit in a credible, public hospital). I have seen a person speak after not being able to speak from suffering a stroke (they spoke the afternoon after receiving a treatment in the late morning). I have had an elderly person in a body cast with multiple fractures. The ‘professionals’ told us that rate of healing was sped up by about a month. I have seen the complete elimination of symptoms of PTSD; anger & resentment disappear; numerous healings of emotional hurts so that people can still have the memories but the emotional ‘tug’ is gone. Pretty powerful snake oil. What the author failed to mention is that the pharmaceutical companies acknowledge that their meds can only treat symptoms but cannot get to the root of any problem and that all synthetic meds cause side effects of their own which are harmful to a certain degree to people. True, they do have their validity but to discount the validity of anything that is not of an allopathic mindset is to eliminate systems that could be of help to people needing medical treatment. The author needs to face her own biases before passing judgment on something she does not truly understand or appreciate the positive effect that these methods can have. To suggest that the biochemical is the standard by which all other methods are called ‘CAM’ is arrogant.

  2. eli says:

    if reiki works it works byt he principle that outside support can infuse another human with belief enough to heal themselves,,reiki just is the process of transfering that belief from one to another among us..we all rely on subjective diagnosis at times…sometimes you just need someone to say you are looking well and usuallywhen you are sick people naturally do that… reiki is a formalization of something we do naturally…you can also train yoruself to believe you are well and tap intoyour natural powers of healing which are very strong in all of us …the problem people have with modern medicine is although it treats the disease more effectively it doesnt do what a dog does it doesnt go up and lick your face and stir the natural healing in a person… and it does what it does with far too much arrogance. just in the same way that religions have a monopoly on the power of prayer when we all know that we dont need to believe in god to do some good intentions for the day and feel good about it same with reiki..and all healers…they are peddling something that we all have inside ourselves..the ability to heal ourselves when we tap into that by lessening our stress and just having positivity and good relaxed healthy open channels for our bodies hormones to flow and work at healing… the thing is some people have so little self belief that they need to pay someone else to give them that healing and i think if reiki healers really believed in the powers of healing they wouldnt charge vast amounts fo rit and if they did they would at lest point out to people that they are capable of doing it for themselves when they get stronger..even teaching people to do that for themselves is very humane but profitting from it seems immoral. it would be like a doctor charging you for pouring you water from your own tap… you can do that yourself… its the things that cant be taut that are charged for the vast array ofequppment gone into researching a drug which acts objectively on a parasite or disease…cancer being one which is very much a part of us as opposed to an outside agent is tricky and i think there is a lot of subjective helaing to be done with cancer because the cancer is under our jurisdiction the body can fight it from within and balance out the body also comas and other non agent reliant diseases ie they dont come from virus or bacteria and even these have intrinsic relationships with our own bodes immunity so theres some subjectivity there that if you can allow the system to function at a high rate of healing my believing in a pill or just directing your energy and blood flow to the core of the body the organs will function all the better to fight an agent causing disease.. ther eis very much need for the placebo effect healing in our medical lives..but people who profit from it give the whole area a bad name just as the very rich vatigan give a bad name to anyon ewho believes in charity… but people who dont have money to throw away will benifit from medicine rather than cad because cad occurs in us naturally we all give healing to our loved ones and if we believed a little more in our own powers of healing than we do in medication and potions we would only need to buy real medicine for real complaints. im getting a back ache typing but i have the power to relieve my own neck pain without going to a chiropractor and paying him 100 dollars just to let me believe in myself enough to relax and feel myself thrive… an organism needs positivity and thats what these people are selling even though its free for anyone on earth.

Get eSkeptic

Science in your inbox every Wednesday!

eSkeptic delivers great articles, videos, podcasts, reviews, event announcements, and more to your inbox once a week.

Sign me up!

Donate to Skeptic

Please support the work of the Skeptics Society. Make the world a more rational place and help us defend the role of science in society.

FREE Video Series

Science Based Medicine vs. Alternative Medicine

Science Based Medicine vs. Alternative Medicine

Understanding the difference could save your life! In this superb 10-part video lecture series, Harriet Hall, M.D., contrasts science-based medicine with so-called “complementary and alternative” methods. The lectures each range from 32 to 45 minutes.

FREE PDF Download

Top 10 Myths of Terrorism

Is Terrorism an Existential Threat?

This free booklet reveals 10 myths that explain why terrorism is not a threat to our way of life or our survival.

FREE PDF Download

The Top 10 Weirdest Things

The Top Ten Strangest Beliefs

Michael Shermer has compiled a list of the top 10 strangest beliefs that he has encountered in his quarter century as a professional skeptic.

FREE PDF Download

Reality Check: How Science Deniers Threaten Our Future (paperback cover)

Who believes them? Why? How can you tell if they’re true?

What is a conspiracy theory, why do people believe in them, and why do they tend to proliferate? Why does belief in one conspiracy correlate to belief in others? What are the triggers of belief, and how does group identity factor into it? How can one tell the difference between a true conspiracy and a false one?

FREE PDF Download

The Science Behind Why People See Ghosts

The Science Behind Why People See Ghosts

Do you know someone who has had a mind altering experience? If so, you know how compelling they can be. They are one of the foundations of widespread belief in the paranormal. But as skeptics are well aware, accepting them as reality can be dangerous…

FREE PDF Download

Top 10 Myths About Evolution

Top 10 Myths About Evolution (and how we know it really happened)

If humans came from apes, why aren’t apes evolving into humans? Find out in this pamphlet!

FREE PDF Download

Learn to be a Psychic in 10 Easy Lessons

Learn to do Psychic “Cold Reading” in 10
Easy Lessons

Psychic readings and fortunetelling are an ancient art — a combination of acting and psychological manipulation.

Copyright © 1992–2017. All rights reserved. The Skeptics Society | P.O. Box 338 | Altadena, CA, 91001 | 1-626-794-3119. Privacy Policy.