Skeptic: Examining Extraordinary Claims and Promoting Science Skeptic: Examining Extraordinary Claims and Promoting Science

top navigation:

Wednesday, July 14th, 2010 | ISSN 1556-5696

eSkeptic: the email newsletter of the Skeptics Society

Share this eSkeptic with friends online. Subscribe | Donate | Watch Lectures | Shop



Follow Michael Shermer on Twitter, Facebook, and TRUE/SLANT
Positively False (book cover)

NEW ON MICHAELSHERMER.COM
Nash Equilibrium, the Omerta Rule, and Doping in Cycling

The Tour de France is underway and it is already shaping up to be one of the grandest and most epic races in the event’s century-long history. This event is so hard it is not surprising that, as usual, allegations and suspicions of doping have surrounded the race even before it began. In this week’s Skepticblog, Michael Shermer explains why race organizations have such a hard time enforcing the rules, and what can be done about it.

READ the post

FOLLOW MICHAEL SHERMER ON TWITTERFacebookTRUE/SLANT

In this week’s eSkeptic, Dr. David H. Voelker reviews Wrong: Why Experts Keep Failing Us — And How to Know When Not to Trust Them, by David H. Freedman.

Dr. David H. Voelker is a Lecturer at Stanford University, where he teaches courses in communication research methods and theory. He is also a research consultant, and is co-author of a best-selling statistics study guide. His interest areas include cognitive processing, the misuse of social science research, and the philosophy of science.


Wrong: Why Experts Keep Failing Us  —  And How to Know When Not to Trust Them (detail of book cover)

How Much Does Being Right Matter?

by Dr. David H. Voelker

THE MARKET FOR BOOKS about how ordinary people make thinking mistakes being fairly saturated (Predictably Irrational, Sway, Nudge), it makes sense that someone would turn the spotlight on a group that’s supposed to mess up less than the rest: experts. Journalist David Freedman walks us through an impressive list of false and conflicting claims made by experts in a variety of fields that really drives home the dubiousness of much — if not most — of what passes for expert wisdom. The book is worth this carefully assembled and annotated collection of dueling truth claims alone.

There’s plenty of blame for our expert misinformation to go around, says Freedman. From respected scientists to financial wizards to self-appointed relationship gurus, people whom we credit with specialized knowledge conduct sloppy research, suppress disconfirming data, and leap to unwarranted conclusions. Journalists oversimplify and misrepresent study findings. Bad advice thrives in part because the public demands easy fixes that are “resonant, provocative and colorful.”

We would expect Wrong to cover famous cases of expert fraud like the South Korean human embryo cloning scandal, and it does. Most of the expert errors documented here, however, are not intentional, but originate in the cognitive biases to which everyone is prone. Like the rest of us, experts have sharper eyes for data that supports their hypotheses, claim to have started out looking for what they eventually found, and play to their employers’ metrics (research funding agencies). Nor are peer review and other forms of self-regulation much of a remedy: Thomas Kuhn showed some fifty years ago how the practices of scientific communities reinforce and perpetuate prevailing paradigms.

It would be unremarkable to learn that experts sometimes make mistakes, but if Wrong is right, the magnitude of the problem is much greater than most people suspect. The findings of two out of every three published medical studies fail to hold up. When you consider that the truth claims of less highly-educated and credentialed experts like the inventor of the latest diet or management fad are on average even less reliable, you realize we’re awash in untrustworthy advice.

Books written to hammer a single point are vulnerable to overstating their case, and Freedman’s expert targets are sometimes damned if they do and damned if they don’t. For example, he criticizes mainstream scientific research for careless procedures and small, unrepresentative samples but later defends junk science because, even though its procedural rigor and signal to noise ratio are typically even lower, it occasionally stumbles upon a nugget missed by the pros. That’s having it both ways. Similarly, he fails to take his own advice against retrospective sense-making when he says the warning signs in a famous case of scientific fraud were “glaringly obvious” when looked back upon.

Freedman knows his task leaves him open to the charge of begging the question: the same lack of certainty he says accompanies expert judgments must also apply to his own assumptions that particular expert claims are wrong. I agree with his defense that, though any individual claim may be mistaken, accumulating and pooling evidence allows us to converge on the truth. A historical progression of geographic maps provides an example: early maps of the world are wildly discrepant, but the shapes of the continents and details in their coastlines gradually converge and stabilize across cartographers over time.

So common are the serious errors catalogued in Wrong by even the most eminent researchers and institutions, though, and so influential are false claims in directing the flow of dollars and in propping up whole industries and reputations, that by the time you reach the simple guidelines at the end of the book for knowing when to suspect that an expert opinion might be wrong, it feels the equivalent of being advised to move a foot inland from the beach to protect against tsunamis.

Wrong: Why Experts Keep Failing Us — And How to Know When Not to Trust Them (book cover)

Wrong’s abundant examples of how experts fail us demonstrate how complex the world is. The contingencies of cause-and-effect relationships can be many and difficult to trace, and good advice in one situation can lead to disastrous results in another.

Unfortunately, directing experts (or the middlemen who report their work) to qualify their truth claims with all of the ways they could be wrong, and publicizing negative findings as well as positive ones, are no solution. That just adds to the pile of stuff we have to sift through, and makes it harder for our brains to fulfill what is perhaps their most important function, and is the main reason we rely on experts in the first place: information reduction; sorting through the noise for the signal. The best we can hope to do (whether expert or layman) is narrow the confidence intervals of our predictions a little, and be wrong a little less often.

On the surface, Wrong is about the untrustworthiness of expert advice, but it has much deeper implications. As William James observed a century ago: “Truth lives, in fact, for the most part on a credit system. Our thoughts and beliefs ‘pass,’ so long as nothing challenges them, just as bank-notes pass so long as nobody refuses them.”

Even though experts might eventually converge on the truth about specific clearly defined causal relationships, across the multitude of main effects and complex interactions that each of us experiences in a typical lifetime, the saying, “the truth will out,” is false. At any given time, a substantial number of our individual and shared beliefs about the causes of those effects are simply wrong. We’re guaranteed to take to our graves false conclusions about why this diet did or didn’t work for us, why that relationship went sour, or whether our grown child wouldn’t have committed spousal abuse if only we’d spared the rod.

The degree to which the truth doesn’t “out” is the latitude that experts — and anyone, for that matter — have to construct social realities with impunity. For the objectively false claims described in Wrong and others we hear every day to have the power they do to launch movements, sell products, determine government policy and distribute social rewards, being right doesn’t matter as much as being accepted.

That, I think, is the real lesson of Wrong.


Skeptical perspectives on the history of science and skepticism
DVD cover The Origins of Skepticism and the JFK Assassination
(DVD $23.95) with Dr. Richard Popkin

In this classic lecture at Caltech, one of the world’s foremost historians of skepticism gives a clear history of skeptical philosophy and presents practical applications of skeptical thinking to modern issues, beginning with the Warren report, of which he was the first prominent public intellectual to be skeptical of the government’s claim that JFK was assassinated solely by Lee Harvey Oswald. This lecture is a classic in skeptical thinking and belongs in the library of all serious skeptics.
READ more…

DVD cover Witches, Spirits and Science: Experimental Science and the Paranormal in 17th Century England
(DVD $23.95) with Dr. Richard Olson

We all know about the paranormal belief in witches among commoners in Early Modern England. But did scientists also believe in witches, witchcraft, and spirits? Historian of science Richard Olson explores the fascinating connections between experimental science and the belief in witches and spirits in the transitional age between medieval superstitions and modern scientific methodology. READ more…

DVD cover False Memory Syndrome and the Recovered
Memory Movement

(DVD $23.95) with Dr. John Hochman

Abuse claims are now considered by many to be nearly epidemic, with some demographics suggesting that nearly 1/3 of all women were abused as children. Are the growing number of stories evidence of widespread crimes against humanity? Or is this a social movement analogous to the witch trials of the Middle Ages, where memory is acquired and the accused are automatically considered guilty? READ more…

15 Comments »

15 Comments

  1. Dimo says:

    We seem to exist in an ever more complex and convoluted world with science, technology, ideology and spiritual beliefs and ideas hurled at us from every direction. One can find a forum for almost anything, and within such pages there is as much rubbish spouted as sense. The printed word has, I believe, over recent decades lost respectability because now almost everyone is creating it, often with scant regard to its veracity.

    The truth seems harder to access and resolve, and indeed with the plethora of conspiracy websites around now, I peronally find I have been questioning ‘facts’ I assumed as truth but which may not be on some level.

    ‘Wrong’ sounds like a fascinating book but I’m not sure I’ll be digesting it. I’m a born sceptic and having been hoodwinked by a so-called ‘truth’ now and again it has made me ever more vigilant. Questioning everything isn’t a way of life that appeals and I’ve occasionally slid down a slope toward lazy thinking, simply accepting what will be will be along with ideas of a harmonic, peaceful and spiritual existence.

    Truth-seeking is hard work and unravelling it can take up too much time and effort for some of us. Perhaps I really have grown lazy, which is why Sceptic.com appeals so much.

    Respect!

  2. Ingrid Eisenstadter says:

    As long as the subject is skepticism, it’s interesting to read in your review of the new book “Wrong” by Dr. David Voelker how the fraud accusations against South Korea’s Dr. Hwang Woo-suk continue. Hwang made international headlines when his papers published in Science in 2004 and 2005 about successfully cloning human embryonic stem cells were shown to contain fabricated data. Until then Hwang was a respected and productive scientist. Given that his chances of getting caught at this fraud were 100 percent as other labs dove on this work to replicate it, there is reason to question Hwang’s involvement. He had a large research team, which included a number of 20-somethings, and there were clearly participants in his work who thought they could get away with fudging data. Yes, Hwang is ultimately responsible for the misrepresentations, but his credentials leave me skeptical about his deliberate, personal involvement in a fraud he could not possibly get away with.

  3. Mike Krpan says:

    During my career in the fitness industry I have encountered all kinds of what people commonly believe to be the secret to losing weight. They usually quote something from a popular book or magazine with some plausible concept of how to easily lose weight. Over the years I’ve learned to be more diplomatic and ease the bottom line on to them: at the end of the day, it comes down to a calories in vs calories out model. Unfortunately, that is too often met with bitterness. What the world could use is a hard-hitting, headlining book that separates truth from error about weight control. Anyone up to the task?

    • Paul Bredderman says:

      I suspect that losing weight is a problem that is individual specific. What works for some might be less effective for others. But, I also think there are come common denominators.

      I have to admit that I come to this latter view from my own personal 73-year experience — a lifetime of being physically active, eating a variety of nutrient-dense foods (high nutrient to calorie ratio), and controlling total calorie intake. There probably are no quick fixes that will obviate this approach, long term.

  4. Stephen L. Gibsson says:

    At the risk of being provocative, does anyone else suspect that the built-in incentive for so much “wrongness” is money? Perhaps even capitalism? (This is not to say there is a better option.) There is great incentive and reward awaiting you if you craft studies into a one-sided case for a health remedy, a novel counterclaim to convention, or even simply to tell the masses what they want to hear—that you’ll solve some unsolvable problem for them, or make them skinny. Doesn’t the entire economic system promote and sustain this type of behavior? Competition over cooperation, even in academia and media, seems like a contributing factor, doesn’t it? Aren’t we rugged individuals just out to make a buck and a name for ourselves, versus seeking truth and building the common good? Aren’t those incentives always going to outshine complex reality and truth?

  5. Mark Goodro says:

    Thanks for the thought provoking review. It does more than simply discredit the content of a book – as so many Skeptic reviews tend to do. It’s also nice to see correct usage of “begging the question.”

    • Bob Pese says:

      Sadly , you get raised eyerows and sneers if you object to the modern usage .

      It’s something that has become so widespread that it is ,by default, correct usage.

      “Immaculate Conception” is another.
      Can somebody direct me/us to a list of such figures of speech which have become correct because of popular misuse.

      I think Dobie Gillis has such a list somewhere

      RJP

  6. Loughlin Tatem says:

    I shift my attention away from the article with a gnawing sense of despair when I am told something is wrong but left with no suggestion whatsoever as to what might be a solution to the problem.It is wrong to tell me that “being right doesn’t matter as much as being accepted” as though it might be right or acceptable to be so.

  7. Bad Boy says:

    I disagree with Voelker’s rejection of “the truth will out” and his opinion that peer-review is not much of a remedy. Voelker is too impatient: the truth will *eventually* out but we may not live to see it.

    As far as peer review … Kuhn said (paraphrased Max Planck?)
    “A new scientific truth does not triumph by convincing
    its opponents and making them see the light, but rather
    because its opponents eventually die, and a new generation
    grows up that is familiar with it.”

    IOW: Peer review does work but not in the way it claims to. The ideals of science are muddied with human nature. We shouldn’t be surprised, as my graduate adviser said: “Remember, scientists are people, too.”

  8. frank says:

    loved the admission about the efficacy of peer review – maybe we will see more tempered language and a little more respect towards “creationists”. it would seem that their reservations about the entrenched bias in mainstream “science” may indeed have some merit.

  9. Tracy Ramsey says:

    If you want to see the true need for works devoted to presenting the scientific method and skepticism in confronting the ignorance all around us, check out the official job search site for Iowa Workforce Development. There are two listings for researching and developing “cold fusion”. I was afraid to even try to apply, for fear it’s simply a MENSA sting operation. Or that Michael Shermer might track me down and beat me senseless with a rolled up copy of “Skeptic.”

    • Marcel says:

      I’m assuming the IWD ads for researching & developing cold fusion is not for developing web applications using Macromedia Cold Fusion…..being a developer, that’s what sprang to mind first :)

  10. Tracy Ramsey says:

    One might certainly hope so, however the listing used the term in lower case letters, and the description of the jobs duties gave no indication that this was anything like a computer application. I can’t help picturing it as simply hanging out in the basement of a fat guy named Toby, taking bong hits and playing Dungeons and Dragons all day, while he tells you about his thesis paper that he’s going to finish,”any day now”. While that might have proven intriguing twenty years ago, I’m afraid my metabolism no longer tolerates a “Skittles” and Dr. Pepper diet.

  11. Kathryn Bay says:

    Here are two two quotes from Charles Darwin, which not only show his ethical stance on scientific investigations, but his deep understanding of the frailties of human cognition. (I cannot guarantee that they are word for word, because I copied them a while ago, and I may have paraphrased.)
    1. To kill and error is a good a service as, and sometimes better than, the establishing of a new truth or fact.
    2. …[I] followed a golden rule, namely that whenever a new observation or thought came across me, which was opposed to my general results, to make a memorandum of it without fail and at once; for I had found by experience that such facts and thoughts were far more apt to escape from the memory than favorable ones.

  12. Bill George says:

    “Experts” needs redefinition, as many will link to knowing absolutes. Perhaps “experts” – at best – can be described by French writer Andre Gide:

    “Believe those who are seeking the truth, doubt those who find it.”

get eSkeptic
our free newsletter

Science in your inbox every Wednesday!

eSkeptic is our free email newsletter, delivered once a week. In it, you’ll receive: fascinating articles, announcements, podcasts, book reviews, and more…


Popular Articles
on skeptic.com

Here are the articles that people have been sharing over the last few days.

Carbon Comic

Carbon Comic (by Kyle Sanders)

Carbon Comic, which appears in Skeptic magazine, is created by Kyle Sanders: a pilot and founder of Little Rock, Arkansas’ Skeptics in The Pub. He is also a cartoonist who authors Carbon Dating: a skeptical comic strip about science, pseudoscience, and relationships. It can be found at carboncomic.com.

Help the
Skeptics Society
at no cost to you!

Planning on shopping at Amazon? By clicking on our Amazon affiliate link, which will open the Amazon Store in your Internet browser, the Skeptics Society will receive a small commission on your purchase. Your prices for all products remain the same, yet you’ll provide essential financial support for the work of the nonprofit Skeptics Society.

amazon.com

See our affiliate links page for Amazon.ca, Amazon.de, Amazon.co.uk, and iTunes links.

FREE PDF Download

Reality Check: How Science Deniers Threaten Our Future (paperback cover)

Who believes them? Why? How can you tell if they’re true?

What is a conspiracy theory, why do people believe in them, and why do they tend to proliferate? Why does belief in one conspiracy correlate to belief in others? What are the triggers of belief, and how does group identity factor into it? How can one tell the difference between a true conspiracy and a false one?

FREE PDF Download

The Science Behind Why People See Ghosts

The Science Behind Why People See Ghosts

Do you know someone who has had a mind altering experience? If so, you know how compelling they can be. They are one of the foundations of widespread belief in the paranormal. But as skeptics are well aware, accepting them as reality can be dangerous…

FREE PDF Download

Top 10 Myths About Evolution

Top 10 Myths About Evolution (and how we know it really happened)

If humans came from apes, why aren’t apes evolving into humans? Find out in this pamphlet!

FREE PDF Download

Top 10 Things You Should Know About Alternative Medicine

Top 10 Things You Should Know About Alternative Medicine

Topics include: chiropractic, the placebo effect, homeopathy, acupuncture, and the questionable benefits of organic food, detoxification, and ‘natural’ remedies.

FREE PDF Download

Learn to be a Psychic in 10 Easy Lessons

Learn to do Psychic “Cold Reading” in 10
Easy Lessons

Psychic readings and fortunetelling are an ancient art — a combination of acting and psychological manipulation.

Copyright © 1992–2014 Skeptic and its contributors. For general enquiries regarding the Skeptics Society or Skeptic magazine, email skepticssociety@skeptic.com or call 1-626-794-3119. Website-related matters: webmaster@skeptic.com. Enquiries about online store orders: orders@skeptic.com. To update your subscription address: subscriptions@skeptic.com. See our Contact Information page for more details. This website uses Google Analytics, Google AdWords, and AddThis tracking software.