Getting honest feedback

On this blog, I've repeatedly argued that peer review is a net waste of time, basically because it doesn't enforce much quality control, it results in long publication times, and reviewers have seemingly gone mad these days with additional experiments. I say "net waste of time", though, because there are some undeniable benefits. Almost any paper will usually benefit from a well-meaning expert looking through the paper, remarking on which explanations are unclear, which claims are oversold (or undersold), and what further data would be nice to include if you have it.

So if we were to eliminate peer review, where would we get this sort of feedback? Well, ideally, through peer review–that is, actually sending the paper (informally) to your peers and asking for their comments. The problem, of course, is that this is one sort of response:

Hey Arjun,
Nice paper! Very exciting result! My suggestion would be to extend out the discussion a bit and cite these few papers (some papers). Good luck!
Science Friend

The problem, as I've documented before, is that nobody has time to read. But I'm lucky to have some good friends who will actually take the time to read a paper and give detailed feedback. For instance, Hyun Youk (postdoc at UCSF) read over a paper Gautham's about to submit and gave us quite detailed and extremely helpful feedback that really strengthened and clarified the paper, like the best peer review ever. I have no idea how to systematize this (I guess our current peer review system is basically that, but anonymous), but it's got me dreaming of a fantasy world where our friends read our papers and love them and make them better and then we post on ArXiV and all get HHMI funding. Sigh. Now to submit this paper and get ready for a war of attrition with the official "peer viewers"...

[Wormpaper outtakes and bloopers] elt-2 RNA level dynamics after heat shock in wild-type and HS::end-1 strains

- Gautham

Second in a series of outtakes and bloopers related to our paper on the relationship of gene expression dynamics and cell division times in the early C. elegans embryo.

elt-2 RNA level dynamics after heat shock in wild-type and HS::end-1 strains

This one is a blooper.

In our paper we perturbed cell divisions using mutants and asked if gene expression would track with those cell divisions. Conversely, it would have been great to modify the levels of transcriptional activators and see if gene expression could start before the cell divisions that normally precede them.

Strains with end-1 under the control of a heat shock promoter are available. end-1 is a well-known activator of elt-2. So we went ahead.

Methods: We obtained strain RJ663bc3 from Joel Rothman's group with the kind help of Yewubdar Argaw. We isolated embryos from a synchronized batch culture and aliquoted them at 25C. We heat shocked each aliquot at a different time, for 5 min. at 34C, and returned them to 25C. These brief but accurate heat-shocks were accomplished by centrifuging the embryos and resuspending them in M9 at the appropriate temperature. All samples were fixed simultaneously. The different aliquots spent between 0 and 40 min. at 25C after heat shock and before fixation.

Results: We did RNA-FISH on all the samples for end-1, end-3, and elt-2. We analyzed the data for each aliquot and obtained the following very interesting-looking result for elt-2 expression after end-1 over-expression by heat-shock:

Each panel shows embryos that were heat shocked within a particular window of their development as indicated in the grey shaded region. The red dots are the elt-2 expression levels for these heat-shocked embryos after they've been returned to 25C. The blue dots are the expression trajectory in wild-type without heat-shock for reference and the dashed vertical lines are E lineage divisions. It was super interesting to see elt-2 starting before the 2E divisions, and that it didn't appear to be sensitive if the heat shock came too early.

Blooper: This result is not in the paper because we see qualitatively similar behavior if we heat-shock wild-type worms, with no end-1 overexpressing transgene. Todd Lamitina alerted me to the critical need of doing that control experiment, which in hindsight I should have done first.

Perspective: What was much worse is that I had reason to know that the experiment would be a failure if I had paused to think about all I'd done. Well before doing this experiment I had looked at wild-type embryos that had been left at 30C (not a viable growth temperature for C. elegans) for an hour and found precocious elt-2 expression in these embryos. I disregarded that result as due to any number of things that could go wrong when you try to grow worms under lethal conditions. Without this confounding problem, the overexpression experiment would have been a valuable addition to the paper no matter what the outcome.

Why we were even looking at gene expression trajectories of wild-type worms under non-viable temperatures is a separate story, and another example of Perhaps there will be something interesting.

[Wormpaper outtakes and bloopers] Expression dynamics in C. elegans mes-2 mutant embryos

- Gautham

Very recently I got into a discussion in which I was being super negative about our tendency as scientists to seek to do "one more experiment," and the feeling that it is possible to increase the impact of work by doing more of it. Frankly, if you look at papers in "top tier" journals we see papers that appear to be a collection of somewhat unrelated results bundled up into a massive effort. So it definitely feels like a necessary evil to survive in academia, and even in our lab you'll hear folks complaining about how "thin" such-and-such paper is. The painful part is that the drive to avoid a "thin" paper often ends up in inconclusive or negative results which are never published.

This morning I woke up thinking that instead of being so damn negative (which is depressing for morale of all involved), we could do something positive and actually put some of that stuff up on the web, something Arjun's been encouraging since he set up the blog but we haven't really followed up with in the group. That way the work wasn't for nothing and even though its not in a journal, maybe google will find it when someone makes a search.

This is the first of a few short posts on experiments that we did in relation to our paper on the relationship between cell division times and gene expression onsets in early development of C. elegans embryos. The core of that paper was completed rather quickly, but we spent quite some time trying to add stuff to it. Here is some of the stuff that didn't make the cut.

Expression dynamics in C. elegans mes-2 mutant embryos

In the midst of working on the project that resulted in the paper I went to a talk by Prof. Susan Mango describing their work on mes-2 mutants (Yuzyuk et al. Dev Cell 2009). MES-2 is a component of Polycomb and its mutation was reported to change the window of developmental plasticity in the embryo. I thought it would be interesting to measure the dynamics of the genes to be featured in our wormpaper in this mutant background, since the paper was all about timing of expression of lineage-specification genes. Perhaps there would be something interesting.

Methods: mes-2(bn11) was the mutant studied by Yuzyuk et al. We got it in strain SS186 from the C. Elegans Genetics Center. mes-2 mutants have a very peculiar phenotype: Progeny of mes-2 homozygous mothers are sterile. We switched the balancer with a GFP balancer by mating with strain MT20110, constructed by Erik Andersen, a friend and collaborator of Arjun's from his postdoc time. I wanted to avoid using a fluorescent worm-sorter, so I ended up manually removing all GFP(+) worms from a small synchronized plate. The remaining worms are either fertile mes-2 first-generation homozygotes or their sterile progeny. Very carefully, we ran those ~1000 worms through a micro-scale version of our worm embryo preparation (which usually works with >10x the amount of worms). We then conducted RNA-FISH as usual. 

Results: In the developmental window we were interested in there were no changes at all in the RNA level dynamics of genes we tested compared to wild type (N2 strain). Below is a figure for our favorite genes: end-1, end-3, and elt-2.

y-axis: Number of RNA counted by RNA-FISH. x-axis: number of nuclei (cells) in the embryo.

Similarly, in our hands, the expression dynamics of elt-7, hlh-1, and elt-1, were nearly identical to N2.

Discussion: Seeing no effect was disappointing, but it doesn't necessarily contradict the Yuzyuk report, since most of its statements on expression level effects deal with 8E or later stages in development. That is right about where our window of interest ended.

Perspective: I did these experiments on the common rationale that Perhaps there would be something interesting. That is a good reason to do an experiment. In fact, that is how all experiments get started.

However, it was a bad reason to withhold developing the manuscript for the actual paper, because this experiment, no matter what the outcome would have been, does not have all that much to do with the question of how cell divisions and expression timing are coordinated. So that should have continued as a parallel rather than an in-series effort. What is very telling is that this result is not in the paper because it was a negative result, but we would have probably found a way to put it in if it was positive. Very few experiments of that sort are related in an honest way to the paper that they are being bundled with. Its just fluff.

But we waited because of that feeling that bundling cool stuff together makes for a more compelling and publishable paper. And because, currently, there is no home for "thin" results, like this little blurb on mes-2. 

Code reuse and plagiarism

Gautham's been doing a bunch of refactoring of our existing codebase for spot counting, in particular using a lot of object oriented design strategies. One of the primary goals is code reuse, which is commonly accepted as A Good Thing in programming. On the other hand, a student in lab has been writing a grant proposal on a topic that is very similar to another grant proposal we have submitted (and yes, we checked with the funding agency, it's okay in this context). But of course, my student felt compelled to have completely new language so that it wasn't "plagiarism", which is A Bad Thing. Which got me wondering: why are we so concerned about plagiarism in this context? Why is reuse of language in one context a worthy goal in and of itself, and complete blasphemy in another? Here's another example: I was at a thesis defense in which the candidate was strong second author on a paper and had included some of the figure legends from the published paper in the figures in her thesis. One of the committee members complained about this, saying that it was important to "write this in one's own words". I agree that it may be a good exercise, but I'm not convinced that using the figure legends is per se A Bad Thing. There just aren't all that many clear and concise ways of writing the same thing.

Maybe it's time to rethink the plagiarism taboo. Just like you can use someone else's computer code (with appropriate attribution), why not be able to use someone else's language? If someone wrote something really well, what's the point in rewriting it–probably less well–just for the sake of rewriting it? Would anyone rewrite a super efficient algorithm just for the sake of saying "I wrote it myself"? All that you need is a good mechanism for attribution, like quotes and links and stuff, you know, like all that stuff we already use in writing. In fact, I would argue that attribution is far more transparent in writing than in computer code, because typically only the programmer sees the code, not the end user. If I run a program, I typically am not looking at the source code, so I don't really know who did what, even if the README file gives credit to other code sources.

One might object to copy-and-paste writing becoming a mish-mash of ill-fitting pieces. First off, this is just bad writing, just as code that just jams together different pieces can be a mess, and will typically require one to code stuff to make things flow together well. But in a world where writing demands are growing daily (certainly the case for me), maybe it's time to consider text-reuse as a good practice, or at least not necessarily a bad one.

What is the best text editor?

Ah, why a post on this, when there are probably already something like 17,000 posts being written on the topic at any given time? Well, it’s a topic we often debate in lab, usually as a cool down from some protracted Python vs. MATLAB vs. Perl vs. R argument. Gautham likes vi, as do many programming junkies. I used to use Emacs for a while, then migrated to a little known text editor called Smultron, and now I use some combination of the IDEs incorporated in MATLAB and R and TextWrangler for everything else. And you know what? It works! I can actually edit text this way! I can save the changes, and the run the program! Amazing. What’s even more amazing is that virtually every text editor does this. The conclusion I came to is that it just doesn’t much matter how many fancy key codes your editor can handle (looking at you, Emacs!), nor how complex or simple the interface is (to a point, and once you get used to it), because unless you’re working with punchcards, your programming is much more likely to be limited by your mind than your choice of text editing program. I used to think more about text editors back when I was in graduate school, and I remember one day stopping by the office of Boyce Griffith, who is one of my academic “brothers” (i.e., we had the same advisor). Now, Boyce is one of the most incredible scientific programmers I’ve ever met, having written literally millions of lines of hard core C and C++ to implement very complex simulations of fluids, including these amazing simulations of the human heart. I saw Boyce was programming away using some fairly primitive text editor, and I was like “Man, you don’t use Emacs or something?” Boyce said “No”, and I said “Why not?” and he said “Whatever, it doesn’t matter.” Amen. In the end, I think the best text editor is, you know, one that edits text.

some thoughts related to "The Frustrated Gene: Origins of Eukaryotic Gene Expression"

- Gautham


Chris forwarded this interesting essay to the group:
The Frustrated Gene: Origins of Eukaryotic Gene Expressionby Hiten D. Madhani.

The arguments are not airtight, but how could they be anyway? A friend of mine has said that these efforts are necessarily a connect-the-dots exercise with the dots spread very far apart. It is not at all easy to figure out how these ancient devices came about. Nevertheless, the presented idea that the complexity of gene regulation was created to temporarily ward off selfish DNA makes more sense than it having arisen primarily to actually enable complexity. I think that the perspective still leaves open the possibility that we've kept all those adaptations because they do enable complexity, but I am nowhere near qualified to judge that for sure.

What is clear is that looking at every aspect of gene regulation in eukaryotes and thinking that it has to create a net benefit to the organism at the present, rather than maybe resulting from a prior bottleneck, a prior arms race with a parasite, or even the defeat at the hands of a parasite, may be a mistake. It is terrifying how natural selection is rather helpless in preventing the spread of even damaging transposons in a sexual population. Biology is supposed to make sense in light of evolution, but it may be that no organism may make sense if we require it to be in all aspects evolutionary optimal.

I recall one of Marc Kirschner's book saying that it was at the time an ongoing embarrassment to evolutionary biologists that they don't have a solid explanation for why sexual reproduction is advantageous. They give us the simple story that it enabled complexity and evolvability, but they themselves are not convinced that the numbers work out, or at least they weren't a decade or two ago. It is hard for us to even make a case for sexual reproduction, and yet some of us are losing sleep trying to explain things like apparent gene redundancy.

It is possible that us larger eukaryotes at least operate in too big a variety of circumstances, and perhaps we compete more on the basis of behavior than on the basis of physiology. Perhaps our environment changes too rapidly, and we reproduce too slowly, to allow us to reach our biochemical evolutionary pinnacle. I've been watching "The Life of Mammals" series from BBC. It seems far easier to explain the extraordinary diversity in behavior and shapes of animals on evolutionary grounds, than it is to explain even basic things like their wildly different DNA content.

Sequencing is the new pcr

- Gautham

I was wondering if the domination of high-throughput sequencing in genome science is an anomaly of the times, so I decided to look at the table of contents of old Genome Research issues. What was Genome Research about before the advent of all this sequencing?

http://genome.cshlp.org/content/2/1.toc
http://genome.cshlp.org/content/1/4.toc

Take a look! It was even more dominated by PCR then than it is dominated by HT sequencing now.