Some tips for how to do science

I was having lunch with Ed Ballister to celebrate his freshly defended thesis (congrats!), and we were thinking: aren't there some best practices to being a good scientist? Like a set of tricks for how to do science? I think maybe many scientists resist this notion because we think is will restrict our creativity. I don't think this is the right way to think about it, though. I still believe in creative inspiration, but the point of our scientific training is to give us the tools to generate a line of inquiry to take a creative idea and shape it to make interesting conclusions. People often "test" this ability in thesis committee meetings, but rarely do we talk about how to develop it. Anyway, sort of abstract, but here are some concrete tips:

1. What would god do? Often times, we pose a question, but our thinking is stifled by constantly getting hung up on experimental constraints. One way to open up thinking that Gautham and I like is to just imagine there are no experimental constraints. If you could measure anything and everything, what is the precise comparison you would do to make the conclusion you want? Then work down to reality from there. For instance, we were interested in thinking about whether cell volume can control gene expression. If we were god, we could imagine just changing cell volume and seeing what happened. Turns out we could actually do something similar to that experiment in real life.
2. Avoid whack-a-mole controls. Sometimes, you end up with a hypothesis for what could be happening, but there are tons of other potential effects you have to control for. This can lead to endless controls, none of which are particularly satisfying. Far better to have a single clever control that definitively answers the question. Here's an example: in the single cell biology field, one of the big questions early on is whether cell-to-cell variability in gene expression is due to stochastic effects or "other global effects". These "other global effects" could be just about anything, like position in cell cycle, amount of ribosome, whatever. You can try and eliminate each one, but it's impossible because of the unknown unknowns. Far better is this beautiful experiment of Michael Elowitz, in which he measured expression from two distinguishable copies of the same gene: ALL unknown unknowns would affect both copies to the same extent, resulting in correlated variability, whereas random variability would be uncorrelated. There was definitely uncorrelated variability, hence expression variability is at least partially due to random effects. Beautiful experiment because of the beautiful control.
3. Write down a draft of your paper as soon as possible. Writing tends to expose lazy arguments and tell you what controls you need that you haven't done yet.
4. Think carefully about the experiments that are only interpretable in one direction. You know, the ones in which if you get one answer, then you can say something awesome, but if you don't get that answer, it's just hard to say anything at all. These experiments can be very useful, but quite often don't work out as intended, and if a project is made up of a lot of these sorts of experiments, well, tread carefully.
5. Whenever you think you figured out what's happening, try to come up with at least 1 or 2 alternative hypotheses. Perhaps one of the biggest issues in the era of scientific "stories" is that we all tend to avoid thinking about alternative interpretations. If you have a hard time coming up with plausible alternatives, then you might be on to something. Another related idea is to take all your data, forgetting about how you got there, and decide whether your main point still fits with most of the data. Sometimes you start down a path with one piece of data, formulate a vision, then try to fit all the rest of the data to that vision, even when the data taken as a whole would probably point in another direction. I've seen several papers like this.
6. Pretend you were a mean and nasty reviewer of your own paper. What would be the key weaknesses? I've found that you usually already know the flaws, but just don't want to think about them or admit them to yourself.
7. Think very carefully about the system you plan to use BEFORE starting experiments. Once experiments get going, they have a momentum of their own that can be hard to overcome, and then you might be stuck fighting the system all the time instead of just getting clean results. These can be big things like choosing the wrong cell line or model organism, or small things like targeting the wrong genes.
8. (Related) That said, it's often hard to anticipate all the ways in which experimental vagaries can torpedo even the most well thought out plans. A scattershot approach can be useful here. Like, if your RNA-seq gives you 20 potential hits, don't just follow up initially on the top hit–try the top 3-5. For example, we had a situation where we tried to measure RNA from this one particular stem cell gene, and for whatever reason, the probe just did not work right. We spent months on it, and it just never worked right. Then we just picked out another 2-3 genes to look at and got great data right off the bat. The point is that some things may work and some may not, and it would be a shame to not follow up on something because the one test case you happened to pick didn't pan out.
9. (Related) My friend Jeff liked to say that there's a strong correlation between projects in which things work well right away and projects that ever work. I think this is spot on, partially because of the principle of Conservation of Total Project Energy. Simply stated, there's only a certain amount of time you can spend on a project before you just get sick of it, and if you spend forever for example getting the basic assay to work, then you just won't have the energy to do cool stuff with that assay. If it works right away, though, then you still have the energy left to use the assay in interesting ways.
10. Avoid salami-slicing experiments. There is a strong tendency to keep doing experiments that you know will work, perhaps with very incremental changes. These experiments typically don't tell you very much. This comes from the fear of the unknown. Fight against that tendency! What is the experiment that will give you the most information, that will decisively eliminate plausible alternatives? Chances are you might already know what that experiment is. Just do it!

Anyway, these thoughts are helpful. Please do comment if you have any additional ideas. I would love to hear them and will happily cull them together into a follow up post.