Reality-based science

Most of the published biological literature is wrong.  It's a perhaps less than well known fact, but it is a fact nonetheless: estimates from industry are that around 75%+ of published results are wrong, and our own experience in lab backs that up.  I personally think the reasons for this range from black to white, with black being outright fraud (which I think is relatively rare), and white being honest mistakes or misinterpretations of data (probably reasonably common).  Then there are all those things in the gray area, including selective exclusion of results that don't fit the "story".  If I had to guess, I would bet that most of the reasons why results don't replicate fall into this gray zone.  There are lots of potential reasons for these sorts of problems, funding, pressure for positive results, etc.  Perhaps it is just our own humanity that gets in the way.

So what to do about it?  I've read about people who have suggested two options, the carrot and the stick.  The stick would be that somebody has to reproduce every study, and you are somehow shamed or shunned or something like that if you are caught out.  Trouble is, who's going to do all this reproduction?  That's a lot of time and effort.  Then there's the carrot.  I like this a bit better.  Here, if you want, you can submit your study for verification.  If it verifies, then you get some sort of seal of approval.  Over time, if this idea catches on, then you would basically have to submit your work for verification for anyone to believe it, and the carrot turns into the stick.

Meanwhile, back in reality, I think it's very unlikely for any of this to ever happen.  First off, who's going to pay for it?  And is it worth paying for?  In a way, pharma companies who actually need their stuff to really work are the ones paying for it now–they try to replicate the experiments they need, finding that the majority (if not the vast majority) are wrong, and then follow up on the ones that prove reproducible.  It's perhaps not ideal to have the literature cluttered up with all these wrong results, but what's the alternative?  Plus, just because something is not reproducible in someone else's lab doesn't mean that it's wrong per se.  Often times, the issue is that biological systems just behave differently for reasons we don't understand, giving different results in different labs on different days.  I think we've all had those "must be something in the water" moments.  Umm... don't have much useful to say about that... :)

I think that another option is to train our students to better sort out the good stuff from the bad stuff.  This is so difficult!  People commonly note that a well-trained student can quickly rip almost any paper to shreds.  Which is true and counterproductive.  I've found that one strategy that has worked for us is to start with a basic question and then look up the papers that help to establish a system in which to answer said question.  These papers tend to be (but are not always) less high profile, but they also seem to be at least somewhat more reproducible than high-profile papers, so you can actually build off them.  So many times, I feel like successful projects are really based on a well-thumbed handful of solid, detailed, technical papers.

The other thing to do is to surround yourself with zero bullshit people who will challenge your assumptions at every step.  It can be hard to deal with at times because doing science is just so generally crazy and risky, but it will make you a better scientist.  And a reality-based scientist.