Friday, May 22, 2015

RNA doesn't correlate with protein? Huh?

tl;dr: I don’t know why people say that RNA doesn’t correlate with protein. There are different contexts to this question, and some recent experiments may make the question a bit confusing, but overall, I’m pretty sure that most of the time, if you increase the amount of RNA for a given gene, you will end up with more of the protein encoded by that gene. I’m sure there are counter-examples, though–if you know of any, please fill me in.

In our group, when we present work on RNA abundances, we are often faced with the question: “Well, what about the protein?” (fair enough). This is usually followed by the statement “Because of course it is well known that RNA doesn’t correlate with protein.” Umm, what?

I have to say that I’m a bit puzzled by this bit of apparently obvious and self-evident truth. I thought that most people accept that the central dogma of DNA to RNA to protein is a pretty solid fact in most cases. So… if you have more RNA, that should lead to more protein, right? Shouldn’t that be the null hypothesis?

Apparently this notion has been around for a long time, though nowadays it is perhaps a bit more conceptually confusing due to a few recent results. Perhaps the biggest one was the Schwanhausser paper in which they compare RNA-seq to mass-spec and show that there is a distinct lack of correlation between mean RNA levels and mean protein levels across all genes (also the Weissman ribosome profiling paper). What this means, on the face of it, is that even if gene A produces more RNA than gene B, then it may be the case that there is more protein B than protein A. Fine. There are differences in protein translation rate and degradation rate, leading to these differences, no surprises there. Plus, Mark Biggins and Allan Drummond make the point that any measurement noise will lead to decorrelation even if things are very correlated, and their reanalyses seem to indicate that the correlation between RNA and protein may actually be considerably higher than initially reported.

The next example that’s a bit closer to home for me is whether RNA levels and protein levels correlate, even for the same gene, across single cells. Here, it gets a bit more complex, and one might expect a variety of behaviors depending on the burstiness of transcription, degradation rate of the RNA and the degradation rate of the protein. Experimentally, there are some cases in which the RNA and protein of a particular gene do not correlate in single cells (Taniguchi et al. Science 2010 is a particularly good example). This may be due to long protein half-life, which effectively smooths over RNA fluctuations. In our PLOS 2006 paper (Fig. 7), we showed that there can be a strong correlation between RNA and protein when the protein degrades fast, and that correlation goes down a lot when the protein degrades more slowly.

And of course there’s the whole world of post-translational modifications, like during the cell cycle, etc., in which protein activity and potentially levels change independent of transcript abundance. Well, dunno what to say about that, I’m biased to just think about RNA. :)

Nevertheless, overall, I think it’s pretty safe to assume most of the time that if you increase RNA abundance for a particular gene, you will end up with more of the encoded protein. I think that should be the null hypothesis. If anyone knows of any counterexamples, please let me know.

Oh, and by the way, in case you’re wondering, transcription also correlates with RNA.

Sunday, May 10, 2015

Retraction in the age of computation

tl;dr: I’ve been wondering recently whether we need to reexamine our internal barometers for retraction now that computational analyses are a bigger part of our work in biomedical sciences, in which the line between data and interpretation are somewhat more blurry. I’m not sure what the answer is, but I would definitely lean on the side of not retracting because of the stigma of “shady data” associated with retractions.

I started thinking about this because I saw Yoav Gilad’s reanalysis of some previous expression profile data and showed that the “interesting” finding went away after correcting for batch effects. Someone on Twitter asked whether the paper should be retracted. Should it?

I grew up with the maxim “Flawed data, retract; flawed interpretation, don’t retract”. I think that made a lot of sense. If the data themselves are not reproducible (fraudulent or otherwise), then that’s of course grounds for retraction. Flawed interpretations come in a couple varieties. Some are only visible in hindsight. For example “I thought that this band on the gel showed proof of XYZ effect, but actually it’s a secondary effect due to ABC that I didn’t realize at the time” is a flaw, yes, but at the time, the author would have been fine in believing that the interpretation was right. Not really retraction worthy, in my opinion. Especially because all theories and interpretations are wrong on some level or another–should we retract Newton’s gravitation because of Einstein?

Now, there’s another sort of interpretational flaw, which comes from a logical error. These can also come in a number of types. Some are just plain old interpretational flaws, like claiming something that your data doesn’t fully support. This can be subtle, like failing to consider a reasonable alternative explanation, which is a common problem. (Flawed experimental design also falls under this heading, I think.) Certainly overclaiming and so forth are rampant and generally considered relatively benign.

Where it gets more interesting is if there is a flaw in the analysis, an issue that is becoming more prevalent as complex computational analyses are more common (and where many authors have to essentially trust that someone did something right). Is data processing part of the analysis or part of the data? I think that puts us squarely in the grey zone. What makes it complex is the interplay between the biological interpretation and the nature of the technical flaw. Here are some examples:
  1. The one that got me thinking about this was when Yoav Gilad reanalyzed some existing expression profiles from human and mouse tissues. The conclusion of the original paper was that human and mouse profiles clustered together, rather than by tissue (surprise!), but upon removing batch effects, one finds that tissues cluster together more tightly than species (whoops!). Retraction? Is this an obvious flaw in methodology? Would it matter whether people figured out the importance of batch effects before or after it was published? If so, how long after? I would say this should not be retracted because these lines seem rather arbitrarily drawn.
  2. Furthermore, if we were to retract papers because the analysis method was not right, then we would go down a slippery slope. What if I analyze my RNA-seq using an older aligner that doesn’t do quite as good a job as the newer one? Is that grounds for retraction? I’m pretty sure most people would say no. But how is that really so different than the above? One could say that in this case, there is little change in the biological conclusion. But there are very few biological conclusions that stand the test of time, so I’m less swayed by that argument.
  3. Things may seem more complicated depending on where the error arises. Let’s take the case of RNA/DNA differences as reported by sequencing, which was a controversial paper that came out a few years back. Many people provided rebuttals with evidence that many of the differences were in fact sequencing artifacts. I’m no expert, but on the face of it, it seems as though the artifact people have a point. Should this paper be retracted? Here, the issue is allegedly a flaw in the early stages of the analysis. Does this count as data or interpretation? To many, it feels like it should be retracted, but where’s the real difference from the two previous examples?
  4. I know a very nice and influential paper in which there is a minor mathematical error in a formula in part of the analysis method (I am not associated with this paper). This changes literally all the results, but only by a small amount, and none of the main conclusions of the paper are affected. Here, the analysis is wrong, but the interpretation is right. I believe they were contacted by a theorist who pointed out the error and asked “when will you retract the paper?”. Should they retract? I would say no, as would most people in this case. Erratum? Maybe that’s the way to go? But I am somewhat sympathetic to the fact that a stated mathematical result is wrong, which is bad. And this is a case in which I’m saying that the biological conclusion should trump the analysis flaw.
Overall, I think the issue of how to deal with problematic papers in which errors involve sometimes murky computational and analytical methods is a difficult one, and I would say that it’s maybe worth figuring out what our standards are. I think the real question is whether computational processing of data is part of the data or part of the interpretation, and I think there are reasonable cases to be made either way. It’s tricky and slightly different than with experiments. If someone does a crappy experiment (like used the wrong buffer), then those data would be marked as irreproducible, and thus could be subject to retraction. If the computational pipeline is documented but has a bug, then technically it’s replicable, if not reproducible. So maybe one way forward is to say that bugs are retractable but methodological flaws are not?

I realize this is a pretty high bar for retraction. For me, that’s fine because, practically speaking, I think it’s far better to just leave flawed papers in the literature. Retractions in biomedical science come with the association of fraud, and I think that associating non-fraudulent but flawed papers with examples of fraud is very harmful. Also, perhaps the data is useful to someone else down the road. We wouldn’t want the data to be designated as “retracted” just because of some mistake in the analysis, right? But this also will depend on what point the data is considered data? For instance, let’s say I used the wrong annotations to quantify transcript abundance per gene and report that data. So then the data is flawed. But probably the raw reads are fine. Hmm. Retract one and not the other?

Anyway, I think it’s something worth thinking about.

Update, 5/12/2015: Lots of interesting commentary around this, especially in the case of the Gilad reanalysis of the PNAS paper. Leonid Kruglyak had a nice point:



Sounds reasonable in this case, right? I still think there are many situations in which this distinction is pretty arbitrary, though. In this case, the issue was that they didn’t watch out for batch effects. Now, once people realized that batch effects were a thing, how long does it take before it’s considered standard procedure to correct for it? 1 year? 2 years? A consensus of 90% of the community? 95%? And what if it turns out 10 years from now that the batch effect thing is not actually a problem after all and the original conclusion was valid? These all sound less relevant in this instance, but I think the principle still applies.

Great point from Joe Pickrell:



I really like the idea of just marking papers as wrong in the comments, perhaps accompanied by making comments more visible. (A more involved version of this could be paper versioning.) In this case, the data were fine, and were the paper retracted, then nobody could do a reanalysis to show that the opposite conclusion actually holds (which is also useful information).

Saturday, May 9, 2015

Thoughts on taking my first class in over a decade

Some folks in our lab (including myself) have embarked on a little experiment, which is semi-informally taking a machine learning class this summer. We’re taking a machine learning class over the summer in a self-directed manner, including doing all the homeworks. The rules are that the people who don’t do the homework have to pay for the lunches of the people who do do the homework. So far, everyone’s paid for themselves. For now… :)

Anyway, this is the first class I’ve taken in well over 10 years (although I’ve taught a bunch since then), and I’m enjoying it immensely! It also feels very different than when I took classes in the past. Firstly, I’m definitely slower. I’m taking way longer to get through the problems. At least partly, I think this is because my brain is not quite as quick as it used to be, for sure. Not sure if that’s just from having a lot of distractions or lack of sleep or just the aging process, but it’s definitely the case. Lame.

Also, I’m slower because my approach to every question is very different than it used to be. When I was an undergraduate taking a bunch of classes, a lot of the time I was just trying to get the answer. Now, with a lot more experience (and a very different objective function), I’m far less concerned with getting the right answer, and so I of course spend a lot more time trying to understand exactly how I arrived at the answer.

More interestingly, though, is the realization that beyond just trying to understand the answer, I’m also spending a lot more time trying to understand why the professor asked the question in the first place. For instance, I just worked through an example of a decision tree and entropy, and while I think my earlier self would have just applied the formulas to get the answer, now I really understand why the problem was set up the way it was and why it’s trying to teach me something. This is something I think I’ve come to appreciate a lot more now that I’ve taught a few courses and have designed homework and exam questions. When I write a question, I’m usually trying to illustrate a particular concept through an example (though I typically fail). As a student, I think I typically missed out on these messages a lot of the time both because I was more concerned with getting the answer and because I didn’t have the context in which to understand what the concept was in the first place. Now, I’m purposefully trying to understand why the question is there in the first place from the very get go.

(Note: it’s really hard to devise questions that reveal a concept to the student. Lots of reasons, but one of them is that I feel like concepts come across best through interaction. Problems for classes, though, typically have to be well defined with clear statements and solutions. In a way, that’s the worst way to get a concept across. Not sure exactly what the right way to do this is.)

Another thing I’ve noticed is that every mathematical operation I perform, from doing an integral to inverting an equation, seems far more meaningful than it used to. I think it’s because I feel like I have a much deeper understanding of why they come up and what they mean. That makes computations a bit slower but far more purposeful (and with less time spent on fruitless directions).

Which leads to another point, which is that I tend to make fewer mistakes than I used to, especially of the silly variety. I think this is because in our research, a mistake is a mistake, silly or not, and having the right answer is the only one that matters. So I’ll take it slow and get it right more often than before, which is a somewhat amusing change from the past.

Anyway, overall, a really fun experience, and one that I highly recommend if you haven’t taken a class in a while.

Friday, May 1, 2015

Can I just normalize expression levels by GAPDH?

tl;dr: Depends on context. Probably yes in many instances, but there are definitely situations where you can’t. And beware of global changes in transcription–may just be volume effects.

Now that Olivia’s paper is out (slidecast, full text), thought I’d write a bit about the time-honored practice of normalizing gene expression by GAPDH. A bit of context: when people did RT-qPCR (remember that?) on bulk RNA isolated from, say, cells with and without drug, the question would arise as to how to normalize the measurement by number of cells, differences in RNA isolation efficiency, etc. The way people normally do this in a practical sense is by dividing by the expression of housekeeping genes like GAPDH, which we assume is roughly the same per cell in both conditions. This is of course an assumption, and one which is most definitely broken in some situations.

The plot thickened around 10 years ago, when people started making measurements showing that absolute transcript abundances can vary dramatically from cell to cell, even for housekeeping genes like GAPDH. So how should you normalize single cell data?

Olivia’s paper provides some answers, but also opens up more questions. One of the principal findings (also see this paper by Hermannus Kempe in Frank Bruggeman's group) is that transcript abundance roughly scales with volume. What this means is that bigger cells have more transcripts, and that while the number of, say, GAPDH mRNA can vary a lot from cell to cell, the concentration varies far less. This holds fairly globally. So what this means is that if you normalize by GAPDH, you are pretty much normalizing by the total (m)RNA content of the cell. In the case of single cell RNA-seq (will write up a comparison of that later), you are essentially also normalizing by total mRNA content. Thus, if you are interested in the concentration of your particular mRNA, this is a reasonable thing to do.

There are a couple of wrinkles here. First, one observation we made was that most of the mRNA we looked at had a higher concentration in smaller cells than in larger cells. It was not as wide as the volume variation, but it could go as high as 2x. We’re not sure of the origin of the effect, and it is possible that there’s some systematic error in our measurement that leads to this (although we really tried a lot of different things to discount such possibilities). In any case, it’s something to consider, especially if you want to be very quantitative.

Another wrinkle is that there are definitely situations we’ve encountered when GAPDH mRNA concentration itself can change. This can happen both homogeneously across the entire population, or even within single cells–in one project we’ve been working on, we see some cells with very high GAPDH transcript abundance right next to cells with very low GAPDH transcript abundance. What to do? If you’re doing sequencing, I think that adding some spike-in controls to help normalize by the total number of molecules could help. Or just do some RNA FISH to get a baseline… :)

Finally, I think it’s really important to carefully consider the directions of causality when making claims about global changes in transcription. Olivia’s heterokaryon experiments clearly show that increasing cell volume/cellular content can directly lead to increased transcription. What that means is that if you make a perturbation and then see a global change in gene expression, it may be (in fact, very well likely is) that the perturbation is somehow causing a cell volume change, which then can result in a proportional global change in transcription. We have seen this very clearly in a number of cases.

Another point is that it really depends on context.  We have a recent example in which absolute expression of a secreted protein remains constant, but the cell volume (and hence GAPDH) expression increases dramatically. So what matters, concentration? Absolute amount? It is secreted, and these cells are living in a primarily acellular environment, so the total secreted proteins presumably depends on the absolute number of molecules rather than the concentration. I think it's all a question of context. Which is of course a complete cop-out, I know... :)

Coming soon: description of a comparison of single cell RNA seq and RNA FISH.

Monday, April 13, 2015

My favorite moment from when I was in grad school was when…

No doubt there’s a lot of bitterness out there about graduate school these days. There’s a steady drumbeat of despair about getting jobs, dealing with the frustrations of failed projects, the pain of publishing, all amid the backdrop of decreased funding. Here are a couple of examples I just saw:
Reading that last one had me nodding in agreement–there are indeed many tough times in grad school. But wait, weren’t there a lot of good ones, too? In fact, looking back at it, grad school was one of the happiest times in my life. And there were many great moments I will never forget. Here are a few of mine:

  • I remember the very first time I saw single molecule RNA FISH spots in the microscope, which came after months of optimization (i.e., messing around). There they were, super bright and unmistakable! I ran and got my advisor Sanjay, who was all smiles.
  • Talking about conspiracies, both scientific and political, with Musa.
  • The first time I saw an endogeneous RNA via RNA FISH (instead of the transgenic RNA spots from before). I felt like I was at the beginning of something very cool, like there were endless possibilities ahead of me. Also wish I had figured it out a couple years earlier... :)
  • When I felt like I had finally figured out cloning with a long string of flawless ligations (winning streak since broken, by the way!).
  • Writing a super complicated (for me) simulation directly in C (implicit method with Newton-Raphson for solving a non-linear PDE in 3D, I think) all in one go and having it work perfectly the first time I compiled it. Yes!
  • When I was feeling like nothing was working and my project was hopeless, and I walked into Sanjay’s office and talked to him for a half an hour, and came out feeling like a million bucks.

I bet many of you have a few of these too, so please leave them in the comments. Would be nice in this particular day and age to have a list of reasons reminding us why grad school might not be so bad after all.

Sunday, April 12, 2015

Why is everything broken? Thoughts from the perspective of methods development

I don't know when this "[something you don't like] is broken" thing became a... thing, but it's definitely now a... thing. I have no real idea, but I'm guessing maybe it started with the design police (e.g. this video), then spread to software engineering, and now there's apparently 18 million things you can look at on Google about how academia is broken. Why are so many things seemingly broken? I think the answer in many cases is that this is the natural steady-state in the evolution of design.

To begin with, though, it's worth mentioning that some stuff is just broken because somebody did something stupidly or carelessly, like putting the on/off switch somewhere where you might hit it by accident. Or the "Change objectives" button on a microscope right next to other controls so that you might hit it accidentally while fumbling around in the dark (looking at you, Nikon!). Easy fodder for the design police. Fine, let's all have a laugh, then fix it.

I think a more interesting reason why many things are apparently broken is because that's in some ways the equilibrium solution. Let me explain with a couple examples. One of the most (rightly) ridiculed examples of bad design is the current state of the remote control:


Here's a particularly funny example of a smart home remote:
Yes, you can both turn on your fountain and source from FTP with this remote.

Millions of buttons of unknown function, hard to use, bad design, blah blah. But I view this not as a failure of the remote, but rather a sign of its enormous success. The remote control was initially a huge design win. It allowed you to control your TV from far away so that you didn't have to run around all the time just to change the channel. And in the beginning, it was just basically channel up/down, volume up/down and on/off. A pretty simple and incredibly effective design if you ask me! The problem is that the remote was a victim of its own success: as designers realized the utility of the remote, they began to pile more and more functionality into it, often with less thought, and potentially pushing beyond what a remote was really inherently designed to do. It was the very success of the remote that made it ripe for so much variation and building-upon. It's precisely when the object itself becomes overburdened that the process stops and we settle into the current situation: a design that is "broken". If everything evolves until the process of improvement stops by virtue of the thing being broken, then practically by definition, almost everything should be broken.

Same in software development. Everyone knows that code should be clean and well engineered, and lots of very smart people work hard to ensure that they make as smart decisions as possible. Why, then, do things always get refactored? I think it's because any successfully designed object (in this case, say, a software framework) will rapidly get used by a huge number of people, often for things far beyond its original purpose. The point where the progress stalls is again precisely when the framework's design is no longer suitable for its purpose. That's the "broken" steady state we will be stuck with, and ironically, the better the original design, the more people will use it and the more broken it will ultimately become. iTunes, the once transformative program for managing music that is now an unholy mess, is a fantastic example of this. Hence the need for continuous creative destruction.

I see this same dynamic in science all the time. Take the development of a new method. Typically, you start with something that works really robustly, then push as far as you can until the whole thing is held together with chewing gum and duct tape, then publish. Not all methods papers, but many are like this, with a method that is an amazing tour-de-force... and completely useless to almost everyone outside of that one lab. My rule of thumb is that if you say "How did they do that?" when you read the paper, then you're going to say "Hmm, how are we gonna do that?" when you try to implement in your own lab.

Take CRISPR as another example. What's really revolutionary about it is that it actually works and works (relatively) easily, with labs adopting it quickly around the world. Hence, the pretty much insane pace of development in this field. Already, though, we're getting to the point where there are massively parallel CRISPR screens and so forth, things that I couldn't really imagine doing in my own lab, at least not without a major investment of time and effort. After a while, the state of the art will be methods that are "broken" in the sense that they are too complex to use outside of the confines of the lab that invented it. Perhaps the truest measure of a method is how far it goes before getting to the point of being "broken". From this viewpoint, being "broken" should be in some ways a cause for celebration!

(Incidentally, one could argue that grant and paper review and maybe other parts of academia are broken for some of the same reasons.)

Saturday, April 11, 2015

Gregg Popovich studied astronomical engineering

I was just reading this SI.com piece about Gregg Popovich, legendary NBA coach of the San Antonio Spurs, and found this line to be really interesting:
By his senior year he was team captain and a scholar-athlete, still the wiseass but also a determined cadet who loaded up with tough courses, such as advanced calculus, analytical geometry, and engineering—astronomical, electrical and mechanical. [emphasis mine]

Now, I'm pretty sure they meant aeronautical engineering, but that got me wondering if there is such a thing as astronomical engineering. Well, Wikipedia says there is something called astroengineering, which is about the construction of huge (and purely theoretical) objects in space. I wonder if Pop is thinking about Dyson spheres during timeouts.