12.10.2013

Replication is the only hope for science

A few days ago, I ran across a blog post called "The Replication Myth: Shedding Light on One of Science’s Dirty Little Secrets."  Published in the Scientific American blog, it argues that 
"...unreliable research [has] been the status quo since the inception of modern science. Far from being ruinous, this unique feature of research is integral to the evolution of science."
Like bad directions from a stranger, this line of argument is well intentioned, partly right, and---in the end---seriously misleading.


I love this picture.  It's from a game called "the Ninja family picnic." 
The post's author (one Jared Horvath, a PhD candidate in neuroscience) argues that science is a messy process.  He begins by citing examples of scientists who might have fudged their data (e.g. Galileo), and discoveries made by accident (Viagra).  Fun stories, good facts.  Nothing wrong here.

Then, surveying this data, Horvath makes a wrong turn:

"There is a larger lesson to be gleaned from this brief history.  If replication were the gold standard of scientific progress, we would still be banging our heads against our benches trying to arrive at the precise values that Galileo reported.  Clearly this isn’t the case."
Horvath is absolutely right that science is a messy process. "An article that is 100 percent valid has never been published."  It's messy because it's carried out by imperfect, sometimes egotistical, and occasionally dishonest people.  But he's absolutely wrong to throw out replication.  In the face of all our human foibles, replication is the saving grace of the scientific method.

Horvath's problem is that he conflates "replication" with "validity."  It doesn't bother me that some experiments fail, or even that some false results get published. What bothers me is that we never even try to replicate most of these results. The biggest problem with today's system of publish-or-perish science is that it discourages replication.  In fields where replication is rare, errors and mistakes are rarely corrected, cheaters are rarely exposed, and the scientific enterprise rarely moves forward.  Replication is the mark of a healthy scientific community.



It's ironic that Horvath supports his argument against replication by using examples of ... replication.  In every one of his cases (Galileo, Darwin, Einstein, Pfizer), some other scientist attempted to replicate the initial result.  As others have pointed out, attempts at replication ares conspicuously missing from some areas of science today.

Bottom line: perfection in science is a myth.  Replication is the thing that keeps the scientific method grounded in fact.  Cut that tether, and the whole practice drifts away into wishful thinking, ballyhoo, and theology.

1 comment:

  1. Have you read Scott Alexander's post on The Control Group is Out of Control? It might be interesting supplemental reading for you. slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/

    ReplyDelete