Project choice: Lean into your strengths

TL;DR: Projects are not entirely good or bad on their own. They have to match the person doing them: you! Be honest with yourself about what your strengths and passions are. Choose a project that is fundamentally aligned with those strengths. Do NOT choose a project that relies heavily on things you are not intrinsically motivated to do. You may be tempted to pick a project to shore up on weaknesses, but don’t. Any project will have aspects that will require you to work on your weaknesses, but a project that is fundamentally aligned with your weaknesses is going to be an exercise in misery.

One of the most common questions I get from new students is how to choose a scientific project. Clearly a super important part of the scientific process, but one that has had a somewhat magical quality to it, as though there is some magic wand that one waves over a set of eppendorf tubes to turn them into a preprint that everyone wants to read. Of course, many scientists have some introspection and insight into their thought processes, and while that has largely been passed on by word of mouth, there have been some wonderful recent efforts to describe project ideation (creativity), selection, and execution (see work from Itai Yanai/Martin Lorsch, Uri Alon, Michael Fischbach, and probably several others I’m missing with apologies).

But I feel like a lot of this discussion has missed one critical feature: you. As in you, the one actually doing the project. Everyone has different strengths and weaknesses as a scientist, or more relevantly, passions and aversions. In my experience, which I’m sure many have shared, it’s the match between the project and the scientist that matters far more than the project on its own.

Why does it matter so much? Here’s my theory. Academic research is a highly unstructured work environment. It is hard to quantify, on a daily, weekly, or even monthly basis, exactly what constitutes “progress”. As such, it relies very strongly on intrinsic motivation. As in, you really have to want to do something in order to put in the sustained effort required to actually do it, because it is very difficult to quantify progress from the outside to help force you to do things you don’t want to do. It is possible to force yourself to do things you don’t want in the short term, but if you are not fundamentally excited to do something, it is very hard to keep yourself motivated to do it in the medium-to-long term.

What does this mean in practice? I think it’s easier to see how it plays out by looking at common failure modes in person-project matching. One common thing I’ve seen is sometimes people feel like they need to build experimental skills even though they are fundamentally more interested in computational work, so they want to work on a project that has a significant experimental component. Then what happens is some version of the following: “I could do this experiment today, but it’s Thursday at 4pm. I’ll do it tomorrow. Oh wait, it’s Friday, now I should probably wait until Monday” and next thing you know a month goes by and the experiment still hasn’t gotten done. Sometimes, if you take the same person and give them a dataset, they’re like “I just need this analysis to finish running by 4pm, then I can run the next step, oh wouldn’t it be cool if XYZ were true, hold on let me try this…”. It’s hard to ascribe these delays or accelerations to any one particular decision, but in aggregate, they have an enormous compounding effect. Same sort of thing the other way around.

By the way, this doesn’t mean that you shouldn’t try things, especially early on. I worked in a lab for a summer after my first year of math grad school basically as an exercise in getting some exposure to experimental work, even though I thought I’d never EVER do it for my actual thesis work. Turns out I had a true passion for experiments. Been trying to lean into that ever since! But you have to continuously evaluate and be brutally honest with yourself about whether you’re doing what you’re doing because you really like it or because you think you should like it. I’ve found graduate students often get caught in the trap of working on what they think they should like instead of what they actually like.

This same reasoning affects choice of advisor, both graduate and postdoctoral, especially the latter. Pick an advisor who can help you build on your strengths, and not someone who specializes in your weaknesses. This is not to say that you can’t have complementary skills—especially for postdocs, it is often very fruitful to combine your skills from your PhD with a set of techniques in the postdoc lab. But if you join a lab where the advisor is a skilled computationalist but you want to do some cutting edge experiments, it must be done with a lot of care. You want to be sure the rest of the environment is strong, because it will be difficult for your advisor to guide you to innovate at the edge of the field given their own strengths and weaknesses. Not to say it can’t be done, but just that it should be done very carefully.

Anyway, all that to say, when choosing a project, make sure it matches your intrinsic strengths and motivations. Research is already hard enough, work on things you like to do!

Notes from Frontiers in Biophysics conference in Paros, episode 1 (pilot): Where's the beef in biophysics?

Long blog post hiatus, which is a story for another time. For now, I’m reporting from what was a very small conference on the Frontiers of Biophysics from Paros, a Greek island in the Aegean, organized by Steve Quake and Rob Phillips. The goals of the conference were two-fold:

1. Identify big picture goals and issues in biophysics, and
2. Consider ways to alleviate suffering and further human health.

Regarding the latter, I should say at the outset that this conference was very generously supported by Steve through the foundation he has established in memory of his mother-in-law Eleftheria Peiou, who sounds like she was a wonderful woman, and suffered through various discomforts in the medical system, which was the inspiration behind trying to reduce human suffering. I actually found this directive quite inspiring, and I’ve personally been wondering what I could do in that vein in my lab. I also wonder whether the time is right for a series of small Manhattan Projects on various topics so identified. But perhaps I’ll leave that for a later post.

Anyway, it was a VERY interesting meeting in general, and so I think I’m going to split this discussion up based on themes across a couple different blog posts, probably over the course of the next week or two. Here are some topics I’ll write about:

Exactly what is all this cell type stuff about

Exactly what do we mean by mechanism

I need a coach

What are some Manhattan Projects in biology/medicine

Maybe some others

So the conference started with everyone introducing themselves and their interests (research and otherwise) in a 5 minute lightning talk, time strictly enforced. First off, can I just say, what a thoughtful group of folks! It is clear that everyone came prepared to think outside their own narrow interests, which is very refreshing.

The next thing I noticed a lot of was a lot of hand-wringing about what exactly we mean by biophysics, which is what I’ll talk about for the rest of this blog post. (Please keep in mind that this is very much an opinionated take and does not necessarily reflect that of the conferees.) To me, basically, biophysics, as seemingly defined at this meeting, as a whole needs a pretty fundamental rebranding. Raise your hand if biophysics means one of the following to you:

1. Lipid rafts
2. Ion channels
3. A bunch of old dudes trying to convince each other how smart they are (sorry, cheap shot intended for all physicists) ;)

If you have not raised your hand yet, then perhaps you’re one of the lonely self-proclaimed “systems biologists” out there, a largely self-identified group that has become very scattered since around 2000. What is the history of this group of people? Here’s a brief (and probably offensive, sorry) view of molecular biology. Up until the 80s, maybe 90s, molecular biology had an amazing run, working out the genetic code, signaling, aspects of gene regulation, and countless other things I’m forgetting. This culminated in the “gene-jock” era in which researchers could relate a mutation to a phenotype in mechanistic detail (this is like the Cell golden era I blogged about earlier). Since that era, well… not so much progress, if you ask me—I’m still firmly of the opinion that there haven’t really been any big conceptual breakthroughs in 20-30 years, except Yamanaka, although one could argue whether that’s more engineering. I think this is basically the end of the one-gene-one-phenotype era. As it became clear that progress would require the consideration of multiple variables, it also became clear that a more quantitative approach would be good. For ease of storytelling, let’s put this date around 2000, when a fork in the road emerged. One path was the birth of genomics and a more model-free statistical approach to biology, one which has come to dominate a lot of the headlines now; more on that later. The other was “systems biology”, characterized by an influx of quantitative people (including many physicists) into molecular biology, with the aim of building a quantitative mechanistic model of the cell. I would say this field had its heyday from around 2000-2010 (“Hey look Ma, I put GFP on a reporter construct and put error bars on my graph and published it in Nature!”), after which folks from this group have scattered towards more genomics-type work or have moved towards more biological applications. I think that this version of "systems biology" most accurately describes most of the attendees at the meeting, many of whom came from single molecule biophysics.

I viewed this meeting as a good opportunity to maybe take score and see how well our community has done. I think Steve put it pretty concisely when he said “So, where’s the beef?” I.e., it's been a while, and so what does our little systems biology corner of the world have to show for itself in the world of biology more broadly? Steve posed the question at dinner: “What are the top 10 contributions from biophysics that have made it to textbook-level biology canon?” I think we came up with two: Hodgkin and Huxley’s model of action potentials, gene expression “noise”, and Luria and Delbrück’s work on genetic heritability (and maybe kinetic proofreading; other suggestions more than welcome!). Ouch. So one big goal of the meeting was to identify where biophysics might go to actually deliver on the promise and excitement of the early 2000s. Note: Rob had a long list of examples of cool contributions, but none of them has gotten a lot of traction with biologists.

I’ll report more on some specific ideas for the future later, but for now, here’s my personal take on part of the issue. With the influx of physicists came an influx of physics ideas. And I think this historical baggage mostly distracts from the problems we might try to solve (Stephan Grill made this point as well, that we need something fundamentally new ways of thinking about problems). This baggage from physics is I think a problem both strategically and tactically. At the most navel-gazy level, I feel like discussions of “Are we going to have Newton’s laws for biology” and “What is going to be the hydrogen atom of the cell” and “What level of description should we be looking at” never really went anywhere and feel utterly stale at this point. On a more practical level, one issue I see is trying to map quantitative problems that come up in biology back to solved problems in physics, like the renormalization group or Hamiltonian dynamics or what have you. Now, I’m definitely not qualified to get into the details of these constructs and their potential utility, but I can say that we’ve had physicists who are qualified for some time now, and I think I agree with Steve: where’s the beef?

I think I agree with Stephan that perhaps we as a community perhaps need to take stock of what it is that we value about the physics part of biophysics and then maybe jettison the rest. To me, the things I value about physics are quantitative rigor and the level of predictive power that goes with it (more on that in blog post on mechanism). I love talking to folks who have a sense for the numbers, and can spot when an argument doesn’t make quantitative sense. Steve also mentioned something that I think is a nice way to come up with fruitful problems, which is looking at existing data through a quantitative lens to be able to find paradoxes in current qualitative thinking. To me, these are important ways in which we can contribute, and I believe will have a broader impact in the biological community (and indeed already has through the work of a number of “former” systems biologists).

To me, all this raises a question that I tried to bring up at the meeting but that didn’t really gain much traction in our discussions, which is how do we define and build our community? So far, it’s been mostly defined by what it is not: well, we’re quantitative, but not genomics; we’re like regular biology, but not really; we’re… just not this and that. Personally, I think our community could benefit from a strong positive vision of what sort of science we represent. And I think we need to make this vision connect with biology. Rob made the point, which is certainly valid, that maybe we don’t need to care about what biologists think about our work. I think there’s room for that, but I feel like building a movement would require more than us just engaging in our own curiosities.

Which of course begs the question of why we would need to have a “movement” anyway. I think there’s a few lessons to learn from our genomics colleagues, who I think have done a much better job of creating a movement. I think there are two main benefits. One is attracting talent to the field and building a “school of thought”. The other is attracting funding and so forth. Genomics has done both of these extremely well. There are dangers as well. Sometimes genomics folks sound more like advocates than scientists, and it’s important to keep science grounded in data. Still, overall, I think there are huge benefits. Currently, our field is a bunch of little fiefdoms, and like it or not, building things bigger than any one person involves a political dimension.

So how do we define this field? One theme of the conference that came up repeatedly was the idea of Hilbert Problems, which for those who don’t know, is a list of open math problems set out in 1900 by David Hilbert, and they were very influential. Can we perhaps build a field around a set of grand challenges? I find that idea very appealing. Although I think that given that I’ve increasingly come to think of biology as engineering instead of science, I wonder if maybe phrasing these questions instead in engineering terms would be better, sort of like a bunch of biomedical Manhattan Projects. I’ll talk about some ideas we came up with in a later blog post.

Anyway, more in the coming days/weeks…

Can we measure science?

I was writing a couple grants recently, some with page limits and some with word limits. Which of course got me thinking about the differences in how to game these two constraints. If you have a word limit, you definitely don’t want to use up your limit on a bunch of little words, which might lead to a bit more long-wordiness. With the page limit, though, you spend endless time trying to use shorter words to get that one pesky paragraph one little line shorter (and hope the figures don’t jump around). Each of these constraints has its own little set of games we play trying to obey the letter of the law while seemingly breaking its spirit. But here’s the thing: no amount of "gaming the system" will ever allow me to squeeze a 10 page grant into 5 pages. While there’s always some gamesmanship, in the end, it is hard to break the spirit of the metric, at least in a way that really matters. [Side note, whoever that reviewer was who complained that I left 2-3 inches of white space at the end of my last NIH grant, that was dumb—and yes, turns out the whole method does indeed work.]

I was thinking about this especially in the context of metrics in science, which is predicated on the idea that we can measure science. You know, things like citations and h-index and impact factor and RCR (NIH’s relative citation ratio) and so forth. All of which many (if not most) scientists these days declare as being highly controversial and without any utility or merit—"Just read the damn papers!" is the new (and seemingly only) solution to everything that ails science. Gotta say, this whole thing strikes me as surprisingly unscientific. I mean, we spend our whole lives predicated on the notion that carefully measuring things is the way to understand the world around us, and yet as soon as we turn the lens on ourselves, it’s all “oh, it’s so horribly biased, it’s a popularity contest, all these metrics are gamed, it’s there’s no way to measure someone’s science other than just reading their papers. Oh, and did I mention that time so and so didn’t cite my paper? What a jerk.” Is everyone and every paper a special snowflake? Well, turns out you can measure snowflakes, too (Libbrecht's snowflake work is pretty cool, BTW 1 2).

I mean, seriously, I think most of us wish we had the sort of nice quantitative data in biology that we have with bibliometrics. And I think it’s reasonably predictive as well. Overall, better papers end up with more citations, and I would venture to say that the predictive power is better than most of what we find in biology. Careers have certainly been made on worse correlations. But, unlike the rest of biomedical science, any time someone even insinuates that metrics might be useful, out come the anecdotes:

• “What about this undercited gem?” [typically one of your own papers]
• “What about this overhyped paper that ended up being wrong?” [always someone else’s paper]
• “What about this bubble in this field?” [most certainly not your own field]

Ever see the movie “Minority Report”, where there are these trio of psychics that can predict virtually every murder, leading to a virtually murder-free society? And it’s all brought down because of a single case the system gets wrong about Tom Cruise? Well, sign me up for the murder-free society and send Tom Cruise to jail, please. I think most scientists would agree that self-driving cars will lead to statistically far fewer accidents than human-driven cars, and so even if there’s an accident here and there, it’s the right thing to do. Why doesn’t this rational approach translate to how we think about measuring the scientific enterprise?

Some will say these metrics are all biased. Like, some fields are more hot than others, certain types of papers get more citations, and so forth. Since when does this mean we throw our hands up in the air and just say “Oh well, looks like we can’t do anything with these data!”? What if we said, oh, got more reads with this sequencing library than that sequencing library, so oh well, let’s just drop the whole thing? Nope, we try to correct and de-bias the data. I actually think NIH did a pretty good job of this with their relative citation ratio, which generally seems to identify the most important papers in a given area. Give it a try. (Incidentally, for those who maintained that NIH was simplistic and thoughtless in how it was trying to measure science during the infamous "Rule of 21" debate, I think this paper explaining how RCR works belies that notion. Let's give these folks some credit.)

While I think that citations are generally a pretty good indicator, the obvious problem is that for evaluating younger scientists, we can't wait for citations to accrue, which brings us to the dreaded Impact Factor. The litany of perceived problems with impact factor is too long and frankly too boring to reiterate here, but yes, they are all valid points. Nevertheless, the fact remains that there is a good amount of signal along with the noise. Better journals will typically have better papers. I will spend more time reading papers in better journals. Duh. Look, part of the problem is that we're expecting too much out of all these metrics (restriction of range problem). Here's an illustrative example. Two papers published essentially simultaneously, one in Nature and one in Physics Review Letters, with essentially the same cool result: DNA overwinds when stretched. As of this writing, the Nature paper has 280 citations, and the PRL paper has 122. Bias! The system is rigged! Death to impact factor! Or, more rationally, two nice papers in quality journals, both with a good number of citations. And I'm guessing that virtually any decent review on the topic is going to point me to both papers. Even in our supposedly quantitative branch of biology, aren't we always saying "Eh, factor of two, pretty much the same, it's biology…"? Point is, I view it as a threshold. Sure, if you ONLY read papers in the holy triumvirate of Cell, Science and Nature, then yeah, you're going to miss out on a lot of awesome science—and I don't know a single scientist who does that. (It would also be pretty stupid to not read anything in those journals, can we all agree to that as well?) And there is certainly a visibility boost that comes with those journals that you might not get otherwise. But if you do good work, it will more often than not publish well and be recognized.

Thing is that we keep hearing these "system is broken" anecdotes about hidden gems while ignoring all the times when things actually work out. Here's a counter-anecdote from my own time in graduate school. Towards the end of my PhD, I finally wrapped up my work on stochastic gene expression in mammalian cells, and we sent it to Science, Nature and PNAS (I think), with editorial rejections from all three (yes, this journal shopping is a demoralizing waste of time). Next stop was PLoS Biology, which was a pretty new journal at the time, and I remember liking the whole open access thing. Submitted, accepted, and then there it sat. I worked at a small institute (Public Health Research Institute), and my advisor Sanjay Tyagi, while definitely one of the most brilliant scientists I know, was not at all known in the single cell field (which, for the record, did actually exist before scRNA-seq). So nobody was criss-crossing the globe giving talks at international conferences on this work, and I was just some lowly graduate student. And yet even early on, it started getting citations, and now 10+ years later, it is my most cited primary research paper—and, I would say, probably my most influential work, even compared to other papers in "fancier" journals. And, let me also say that there were several other similar papers that came out around the same time (Golding et al. Cell 2005, Chubb et al. Curr Biol 2006, Zenklusen and Larson et al. Nat Struct Mol Bio 2008), all of which have fared well over time. Cool results (at least within the field), good journals, good recognition, great! By the way, I can't help but wonder if we had published this paper in the hypothetical preprint-only journal-less utopia that seems all the rage these days, would anyone have even noticed, given our low visibility in the field?

So what should we do with metrics? To be clear, I'm not saying that we should only use metrics in evaluation, and I agree that there are some very real problems with them (in particular, trainees' obsession with the fanciest of journals—chill people!). But I think that the judicious use of metrics in scientific evaluation does have merit. One area I've been thinking about is more nefarious forms of bias, like gender and race, which came up in a recent Twitter discussion with Anne Carpenter. Context was whether women face bias in citation counts. And the answer, perhaps unsurprisingly, is yes—check out this careful study in astrophysics (also 1 2 with similar effects). So again, should we just throw our hands up and say "Metrics are biased, let's toss them!"? I would argue no. The paper concludes that the bias in citation count is about 10% (actually 5% raw, then corrected to 10%). Okay, let's play this out in the context of hiring. Let's say you have two men, one with 10% fewer citations than the other. I'm guessing most search committees aren't going to care much whether one has 500 cites on their big paper instead of 550. But now let's keep it equal and put a woman's name on one of the applications. Turns out there are studies on that as well, showing a >20% decrease in hireability, even for a technician position, and my guess is that this would be far worse in the context of faculty hiring. I've know of at least two stories of people combating bias—effectively, I might add—in these higher level academic selection processes by using hard metrics. Even simple stuff like counting the number of women speakers and attendees at a conference can help. Take a look at the Salk gender discrimination lawsuit. Yes, the response from Salk about how the women scientists in question had no recent Cell, Science, or Nature papers or whatever is absurd, but notice that the lawsuits themselves mention various metrics: percentages, salary, space, grants, not to mention "glam" things like being in the National Academies as proxies for reputation. Don't these hard facts make their case far stronger and harder to dismiss? Indeed, isn't the fact that we have metrics to quantify bias critical here? Rather than saying "citations are biased, let's not use them", how about we just boost women's cites by 10% in any comparison involving citations, adjusting as new data comes in?

Another interesting aspect of the metric debate is that people tend to use them when it suits their agenda and dismiss them when they don't. This became particularly apparent in the Rule of 21 debate, which was cast as having two sides: those with lots of grants and seemingly low per dollar productivity per Lauer's graphs, and those with not much money and seemingly high per dollar productivity. At the high end were those complaining that we don't have a good way to measure science, presumably to justify their high grant costs because the metrics fail to recognize just how super-DUPER important their work is. Only to turn around and say that actually, upon reanalysis, their output numbers actually justify their high grant dollars. So which is it? On the other end, we have the "riff-raff" railing against metrics like citation counts for measuring science, only to embrace them wholeheartedly when they show that those with lower grant funding yielded seemingly more bang for the buck. Again, which is it? (The irony is that the (yes, correlative) data seem to argue most for increasing those with 1.5 grants to 2.5 or so, which probably pleases neither side, really.)

Anyway, metrics are flawed, data are flawed, methodologies are flawed, that's all of science. Nevertheless, we keep at it, and try to let the data guide us to the truth. I see no reason that the study of the scientific enterprise itself should be any different. Oh, and in case I still have your attention, you know, there's this one woefully undercited gem from our lab that I'd love to tell you about… :)

What will happen when we combine replication studies with positive-result bias?

Just read a nice blog post from Stephen Heard about replicability vs. robustness that I really agree with. Basically, the idea under discussion is how much effort we should devote to exactly repeating experiments (narrow robustness) vs. the more standard way of doing science, which is everyone does their own version to see whether the result holds more generally (broad robustness). In my particular niche of molecular biology, I think most (though definitely not all, you know who you are!) errors are those of judgement rather than technical competence/integrity, and so I think most exact replication efforts are a waste of time, an argument which many other have made as well.

In the comments, some people arguing for more narrow replication studies made the point that very little (~0%) of our current research budget is devoted to explicitly to replication. Which got me wondering: what might happen if we suddenly funded a lot of replication studies?

In particular, I worry about positive-result bias. Positive-result bias is basically the natural human desire to find something new: our expectation is X, but instead we found Y. Hooray, look, new science! Press release, please! :)

Now what happens when when we start a bunch of studies with the explicit mandate to replicate a previous study? Here, the expectation is now what was already found and so positive-result bias would bias towards a refutation. I mean, let’s face it, people want to do something interesting and new that other people care about. The cancer reproducibility project in eLife provides an interesting case study: most of the press around the publication was about how the results were “muddy”, and I definitely saw a great deal more interest in what didn’t replicate than what did.

Look, I’m not saying that scientists are so hungry for attention that most, or even more than a few, would consciously try to have a replication fail (although I do wonder about that eLife replication paper that applied what seemed to be overly stringent statistical criteria in order to say something did not replicate). All I’m saying is the same hype incentives that we complain about are clearly aligned with failed replication results, and so we should be just as critical and vigilant about them.

As for apportionment of resources towards replication, I think that setting aside the question as to whether it’s a good use of money from the scientific perspective (I, like others, would argue largely not), there’s also the question of whether it’s a good use of human resources. Having a student or postdoc work on a replication study for years during their training period is not, I think, a good use of their time, and keeps them from the more valuable training experience of actually, you know, doing their own science—let alone robbing them of the thrill of new discovery. Perhaps such studies are best left to industry, which is where I believe they already largely reside.

Some (reluctant) thoughts on postdoc pay

Update 12/7/2016: (first follow up here, second follow up here)

I have generally steered well clear of the issue of postdoc pay, which engenders pretty heated conversations that I'm SO not interested in getting into publicly, but one thing I'm seeing is really bugging me these days: people bragging on Twitter about how much they pay their postdocs above the NIH minimum. Like this:

I don't mean to single these folks out—it just happened that I saw these tweets most recently—but I've seen a few such statements over the last year or so since the announcement that the mimimum for salaried workers would be increased to ~$48K or so (which was just recently reversed).

Why is this irritating? Well, first of all, in this funding climate, and given many labs that have to make many tough choices, it does strike me as a bit arrogant to talk about how much more you can afford to pay than many, many other very well-intentioned scientists. The implication is that people who don't pay as much as you do are paying an abusively low amount, which is I think an unfair charge. For these reasons (and maybe a few others), I just don't think it's really appropriate to publicly talk about how much you pay your people. For the record, I support paying postdocs well, and I think the increase is overall a good idea. My point here will be that there is not an obvious default "right" position on the issue of postdoc pay, and I think it is far more complex than just saying "We should pay postdocs a decent wage."

Indeed, I think the key difficulty is pinning down exactly what we mean by the notion of "decent wage". For instance, in the first tweet above, the PI is from Cambridge/Boston, and the second is from NYC. Now, the proposed federal regulation for starting postdocs is (was) $47,484, and that would apply everywhere. Including, say, Ann Arbor, Michigan (which I choose for no particular reason other than it's home to a major, world-class research institution, but in a relatively affordable location). Now, comparing the cost of living of any two places is tricky, but I found this estimate that Boston is roughly 1.4x as pricey as Ann Arbor (which sounds probably about right). Bragging about paying $60K? Well, shouldn't that be $66K? Live in Cambridge MA instead? No better, $76K. So let's stop crowing about how "decently" the Broad Institute pays, okay?

So, is $60K "fair"? Hmm. From the PI perspective: a Boston PI could say, well my dollars don't go as far, so in a way, doesn't the Michigan PI have an unfair advantage? Then again, the Michigan PI could say hey, why do I have to pay more (relatively speaking) for my postdocs? Why does the Boston PI not have to pay the same effective wages I do? Why should they not have an enforced effective minimum standard pay and have the freedom to pay effectively less?

The motivation of PIs may also matter here as well. The focus in the discussion has been on PIs taking advantage of cheap labor, and that definitely happens. But some PIs may define their mission as training as many scientists as possible, which certainly seems reasonable to me, at least from one point of view. (And I do wonder how often those who brag about paying so much above the minimum have actually had to make the tough choice of turning away a talented postdoc candidate due to constrained funding.)

From the NIH perspective: what is the goal? To get as much science as "efficiently" as possible? To train people? To create a stable scientific workforce? Or to better human health? Should the NIH even allow people in high cost of living areas to pay their postdocs more? Would it be fair to consider this pay scale in grant review, just as other areas of budgets are scrutinized? Does increasing the minimum penalize those who pay the minimum in non-Boston/SF locations unfairly, thus increasing inequity? Or does it provide a general boost for those places, now making them more attractive because their NIH minimum dollars go further? Should the NIH scale the size of grant by cost of living in the area of the host institution? To what extent should the NIH support diversity of locations, anyway?

From the trainee perspective: It's pretty easy for trainees to say that whatever they're paid right now is not fair (though you might be surprised how little many assistant professors make). So for trainees reading this post, let me ask: what would be fair? Okay, maybe now you have a number in your head. Where does that number come from? Is it based on need? Consider: should a postdoc who has a family be paid more? Wait a minute, what about the postdoc without a family? What about immigrants with expensive visa costs? Or potentially families to support in their home country? Moving costs? Commuting costs? Should postdocs be paid more when the institution is in an expensive city? Should postdocs be forced to live further away from the institute to seek more affordable housing? My point is that there is no clear line between necessity and luxury, and wherever that blurry line does get drawn will be highly dependent on a trainee's circumstances and choices.

Or should that number be based on performance? Should the postdoc entering the lab with a flashy paper or two be paid more than the one without? Should a postdoc get a raise every time they publish a paper, scaled by how important the paper is? How many grants it generates? I think it's reasonable to assume that such an environment would be toxic within a lab, but wouldn't the same be true of pay based on personal circumstance, as just discussed above? And isn't such performance-based pay already what's sort of happening at a more global level in flush institutes where PIs can get enough grants to pay well above the minimum?

As you have probably noticed, this post has way more question marks than periods, and I don't claim to know the answers to any of these questions. I have thoughts, like everyone else, and I'm happy to talk about them in person, where nuance and human connection tend to breed more consensus than discord. My point is that reducing all this to a single number is sort of ridiculous, but that's how it works, and so that's what we all have to start from, along with various institutional prerogatives. In the meantime, given how simplistic it is to reduce this discussion to a single number, can we please stop with the public postdoc pay-shaming?

Anti-Asian bias in science

Scientists are a cloyingly liberal bunch. In the wake of this (horrifying) election, seems like every other science Tweet I saw was like

To all my Inuit friends and colleagues: I stand with you. Against fear. Against hate.

Lovely sentiments, for sure, and as a non-white person living in the Philly suburbs at this frightening time, that is welcome. (Although I do wonder who would actually step up if something really went down. Would I? Would I even stand up for myself?)

At the same time, beneath this moralistic veneer, it is of course impossible to deny that there is tons of discrimination and bias in science. Virtually any objective look at the numbers shows that women and under-represented minorities face hurdles that I most definitely have not, and these numbers are backed up with the personal stories we have all heard that are truly appalling. But there is, I think, another less widely-acknowledged or discussed form of discrimination in science, which is discrimination targeted towards Asian scientists.

Asians make up a relatively small (though rapidly growing) portion of the US population. In science, however, they're highly over-represented, making up a large fraction of the scientific workforce. And with that comes a strange situation: a group that is clearly not a small minority, and that is doing well in this highly regarded and respected area, and yet clearly faces bias and discrimination in a number of ways, many of which may be different from those that other minorities face.

First off, what do I mean by Asian? I'm guessing I'm not the only one who feels like I'm checking the "miscellaneous box" when I'm faced with one of these forms and choose "Asian":

I mean, there's a billion Indians and a billion Chinese people EACH out there (not to mention 10s to 100s of millions of other Asian groups), but whatever. Point is, Asians are a diverse group, and I think these different groups face some common and some distinct forms of discrimination. Aside from the various distinctions by ethnic category, there are also distinct forms of bias directed towards Asians coming from abroad as opposed to Asian-Americans. I think all Asians face some measure of discrimination, and in particular, those of East Asian (and within that, Chinese) origin face some of the biggest obstacles.

(I could be completely wrong about this, but I do feel like East Asian scientists face more barriers than South Asians for whatever reason. Part of this may be an matter of numbers: there are simply fewer South Asians in science to begin with. And certainly South Asians from abroad run into trouble, especially a generation ago. That said, as an Indian-American I don't personally feel like I've been on the short end of the stick for racial reasons. Then again, who knows what I'm not hearing, know what I mean? Indeed, I think it's specifically because I'm not Chinese that I've seen mostly anti-Chinese bias, which is what I'll focus on here.)

Exactly what are these barriers? After all, don't the stereotypes of Chinese in the US typically involve words like "diligent", "hard working", "good at math"? Well, I think it's important to realize that it is these very words that implicitly provide an upper limit on what Chinese scientists can aspire to in academia. Consider the following statement I heard from someone (I can't exactly remember the context) that went something like "Oh, they'll just hire a bunch of Chinese postdocs for that, I'm sure." As in "do what they're told", "just labor", "interchangeable", "quiet". Are such sentiments that far from "not independent-minded" or "lacking vision"?

You'd think that these stereotypes may have faded in recent years, and I think that is true to some extent. Then again, take a look at this well-meaning guide from a university in Germany for Chinese/German relationships called "When a Chinese PhD student meets a German supervisor", written by a couple of Chinese PhD students in Germany. I think it actually has a lot of useful things in there, and it would be disingenuous to say that there are no meaningful cultural differences, especially for a foreign student coming to Germany. At the same time, I found some aspects of the guide worrisome:

Through constant discussions, Ming gradually learned when he should obey his supervisor and when he should argue. Ming’s supervisor was very happy when he noticed that the way Ming approached his work had changed and therefore said, “German universities train PhD students to think independently and critically.”

There it is: implicitly, Chinese students don't think independently or critically without extensive German retraining.

And check out this one:

PhD students in Germany are not just students, they often are also researchers and employees at universities. On the one hand, they need to finish their scientific projects independently; on the other hand, they have to teach courses that are assigned by the university or their research groups and they have to do daily organizational work as well. All these tasks require professional qualities. In each research group, every member performs his or her duties according to their contracts.

At the beginning of his PhD, Ming had no plan or agenda at all when he talked to his supervisor, which resulted in aimless and inefficient discussions. After being reminded by the supervisor, Ming began to write agendas for their discussions, but they were always extensive instead of being brief, which made it a laborious task for the supervisor to read. Then the supervisor taught Ming to use bullet points, i.e., to list every question or issue that needs to be discussed with a word or a short phrase.

Right… because I've never had non-Asian students who had these problems with "professional qualities".

I mean, I think this guide is addressing some real concerns and is probably very helpful (check out the part where they describe how to sort garbage like the locals—sounds like someone had a traumatic experience leading to that particular section). But there are long-term consequences to reinforcing the stereotypes of lack of independence, lack of communication skills and the such. Notice how these stereotypes are all about the qualities people think are required for getting to the next level in academia?

Also, this stereotyping is not the only form of bias and racism that Chinese people face in science. Indeed, because the number of Chinese people in science is so large, they must constantly be vigilant about accusations of favoritism and reverse bias. This can come out in particularly nasty ways. For instance, I recently went to a major conference and had a chat with a rather well-known colleague after a meal. As is standard, we spent some time complaining about annoying reviewers, and all of a sudden, my colleague said "And I just KNOW this reviewer is Chinese." The venom with which the word "Chinese" came out of their mouth really took me by surprise, but I'm betting I'm not the only one who's heard that sort of thing, and more than once. Just imagine hearing this kind of talk about any other racial or ethnic group.

In that environment, is it surprising that it is hard for Asian scientists to break through to higher levels in academia? It seems to me that Asians form an under-over-represented class in science: they are a big part of making the scientific enterprise run, but have got plenty of extra hurdles to jump through to get to the next level, with bias working against them on precisely all those extra, conveniently unquantifiable qualities deemed necessary to get, say, a faculty position. My father is an academic, and was pretty sure that he faced racism earlier in his career, though it's hard to pinpoint exactly where and how. I had a recent conversation with a Chinese colleague who told me the exact same thing: he knows its harder for him for a number of reasons, but it's just so hard to prove. It is the soft nature of this bias that makes it so pernicious, which is of course true for other groups as well, but I feel like we don't think about it as much for Asians because they are so visibly over-represented, so we think "What's the problem?".

All this is not to say that there's been no progress. For instance, at the very conference where my colleague lamented their allegedly Chinese reviewer, I noted just how many of the best and brightest PIs in attendance were Asian, including a large number of Chinese and Chinese-American scientists. Indeed, I just visited a university where my hosts were extremely successful Asian scientists, and they so were warm and welcoming, inviting me to dinner at their home together with a few other Asian scientists, all of whom I really admire and respect. At those times, I think the vision of an inclusive, open-minded scientific community is not only possible, but perhaps attainable.

At the same time, I think recent events have shown that these changes do not come for free. It is a cliché, but it is true that we must all fight for these changes and stand against fear and against hate, etc., etc. Great, that's fine and well, and I'm all for it. At the same time, I think it's important to acknowledge that when it comes down to it, social pressures often make it hard to say something in the moment when these situations arise. Looking back at my own experiences, I think I am not alone in saying that I have more regrets about lost opportunities to do or say the right thing rather than proud moments of actually standing up to what I think was wrong. Just saying "we should stand up to bias and discrimination" is very different than providing a blueprint for how to do so.

As such, all moral grandstanding aside, I think there is an interesting question facing us Asians now as a group. Thus far, I feel that Asian scientists have relied on the goodwill of non-Asians to advocate for us, push our careers, make a place for us in science—and to the many, many wonderful scientists who have supported Asians, including myself, a sincere thank you. But it's important to realize that this means, essentially, succeeding on other people's terms. Those terms have generally been favorable to Asian scientists (and non-scientists) so far, but are there limits to Asian success in that model? Do we need to start asserting our rights more aggressively and in a more organized fashion? A postdoc in my lab, Uschi, has vigorously spoken out for postdoc rights here at Penn, and guess what: it makes a difference. I would imagine that advocating for Asians scientists could result in similar benefits. Should this be part of a larger effort to assert Asian rights on a national stage? After all, while relying on the benevolence of kind-hearted non-Asian scientists has worked okay so far in our little science bubble, if we think that general nerdiness and funny accents are going to save us in Gen Pop, well, take a look at what's going on in the aftermath of this election. Maybe it will require concerted, coordinated advocacy to change the policies and bias that make things difficult for foreigners that science in this country relies on, Asian and otherwise.

Gotta say, I felt very weird writing this last paragraph. Does this come across as shrill and ungrateful? Why am I rocking the boat? Making a mountain out of a molehill? Shouldn't we just keep our heads down and focus on our work? These are questions I asked myself as I wrote this as a person who has done well in the system and doesn't really have that much to complain about. But maybe that's just me "being Asian"?

PS: Here's another snippet from the German guide for Chinese students:

The third surprise was that on the same day Ming arrived in Konstanz, the research group threw a welcome party for him and all the group members showed up. At that party, Ming got to know everybody. Besides, there was a discussion about picking a German name for Ming. Based on the group members’ opinions and Ming’s agreement, he was finally named Felix, which indicates optimism and therefore matches his character. From then on, he has had a German name. The thoughtful and warm welcome from his research group touched Ming and he was looking forward to the cooperation with his research group.

Okay, whatever else happens, can we at least agree to stop this forced renaming business?

[Update, 11/20: Apparently, the word Eskimo is now considered derogatory; changed to Inuit, no offense intended.]

The kinship of the arcane

Just had a great visit to University of Calgary, and one of the highlights was meeting with Jim McGhee. It was a feeling I’m sure many of you have had as well—finally meeting someone whose papers have been really influential in your life. In this case, it took me back first to my postdoc, when Jim’s work on C. elegans intestinal development formed much of the basis for work we did on incomplete penetrance. Many of his papers were amongst my most well-thumbed when we were working on that project, and were simply invaluable as we tried to piece our findings together. Then, my fledgling lab’s fate intertwined with Jim’s again when we (meaning Gautham and I) started asking questions about cell cycle and gene expression timing during intestinal development. Gautham is a true scholar, and as this project progressed, we were continually drawn to Edgar and McGhee, "DNA Synthesis and the Control of Embryonic Gene Expression in C. elegans", Cell 1988, a classic paper on this same question.

We would read that paper over and over, delving into each minor point in detail, all the while wondering who this “Lois Edgar” was, marveling at skill and fortitude that must have been required to pull off those seemingly impossible experiments. To give you a sense of what was involved, the question was DNA replication was required for expression of certain genes during development (i.e., is cell cycle somehow a timing mechanism). To answer the question, Lois would take live embryos at precise stages, permeabilize their eggshells through mild smushing or completely removing them via pipetting, then add aphidicolin to inhibit DNA synthesis, and then trace those embryos over time to look for expression of gut specific factors. These are most certainly NOT easy experiments to do, and I (think) I remember my friend John Murray telling us that McGhee said that only Lois Edgar could do those experiments. Gautham also found this really interesting self-profile by Lois in which she talks about her career going to graduate school at age 40 (!). She sounded like a very interesting person to both of us.

So I was sort of nervous to meet Jim. What would he be like? Gregarious and fun? Quiet and bitter? Well, unsurprisingly for a worm person, he was wonderful! We had a great time over dinner talking about various things, including, of course, Lois Edgar, and about the many hours she spent at the microsope watching worms. Also, he talked about how she was a very talented artist. Indeed, the next day, he showed me this lovely rendition of the worm she drew for him many years ago that only an artistic geneticist (or genetically-inclined artist?) could create:

Apparently she has now indeed followed through on her promise to retire from science and go back to art, and makes pottery at her home in Boulder. I actually go to Boulder fairly often, and I’m thinking I should try and meet up with her next time I’m out there.

Which, upon reflection, is a bit odd. After all, how strong can a connection be between two strangers linked by nothing but a model organism and some gene names? I think that’s actually part of the magic of science: at its purest, it’s a kinship, passed through generations, forged by a common interest in arcane details and minor subtleties that perhaps only three or four people in the world know, much less care about.

I also had the opportunity to meet two of Jim’s students, and I’m happy to report that they are doing some very cool science. It has been a while since I’ve thought about intestinal development in worms, but talking with them was like putting on an old pair of shoes—it was nice to talk about old friends like elt-2 and end-3. I could sense that the students enjoyed talking with someone who knew the details of the system to which they have devoted a fair fraction of their waking hours to studying. One of them had a marked up copy of our C. elegans paper out, and I pointed out a minor detail about ectopic expression in hda-1 knockdown worms that is very easy to miss in the paper and might be relevant to their work. In the end, it is people who keep a field alive, speaking to each other imperfectly over the years through dusty pages.

Honesty, integrity, academia, industry

[Note added 7/22/2016 below in response to comments]

Implicit in my last post about reputation in science was one major assumption: that honesty and integrity are important in academia. The reason I left this implicit is because it seems so utterly obvious to us in academia, given that the truth is in many ways our only real currency. In industry, there are many other forms of currency, including (but not limited to) actual currency. And thus, while we value truth first and foremost in academia, I think that in some areas of industry, even those perhaps closely related to academia, the truth is just one of many factors to weigh in their final analysis. This leads to what I consider to be some fairly disturbing decision making.

It’s sort of funny: many very talented scientists I know have left academia because they feel like in industry, you’re doing something that is real and that really matters, instead of just publishing obscure papers that nobody reads. And in the end, it's true: if you buy an iPhone, it either works or doesn’t work, and it’s not really a debatable point most of the time. And I think most CEOs of very successful companies (that actually make real things that work) are people with a lot of integrity. Indeed, one of the main questions in the Theranos story is how it could have gotten so far with a product that clearly had a lot of issues that they didn’t admit to. Is Theranos the rare anomaly? Or are there a lot more Elizabeth Holmes’s out there, flying under the radar with a lower profile? Based on what I’ve heard, I’m guessing it’s the latter, and the very notion that industry cares about the bottom line of what works or doesn’t has a lot of holes in it.

Take the example of a small startup company looking for venture capital funding. Do the venture capitalists necessarily care about the truth of the product the company is selling or the integrity of the person selling it? To me, from academia, I thought this would seem to be of paramount importance. However, from what I’ve been hearing, turns out I was completely wrong. Take one case I’ve heard of where (to paraphrase) someone I know was asked by venture capitalists at some big firm or another to comment on someone they were considering funding. This person then related some serious integrity issues with this person to the venture capitalists. To which the venture people said something like “We hear what you’re saying. Thing is, I gotta say, a lot of people we look at make up their degrees and stuff like that. We just don’t really care.” A lot of people make up their degrees, and we just don’t really care. A number of other people I know have told me versions of the same thing: they call the venture capitalists (or the venture capitalists even call them), they raise their concerns, and the venture people just don’t want to hear it.

Let’s logic this out a bit. The question is why venture capitalists don’t care whether the people they fund are liars. Let’s take as a given that the venture capitalists are not idiots. One possible reason that they may not care is that it’s not worth their time to find out whether someone has faked their credentials. Well, given that the funding is often in the millions and it probably takes an underling half a day with Google and a telephone to verify someone’s credentials, I think that’s unlikely to be the issue (plus, it seems that even when lies are brought to their attention, they just don’t care). So now we are left with venture capitalists knowingly funding unscrupulous people. From here, there are a few possibilities. One is that someone could be a fraud personally but still build a successful business in the long term. Loathe as I am to admit it, this is entirely possible—I haven’t run a business, and as I pointed out in the last post, there are definitely people in science who are pretty widely acknowledged as doing shoddy work, and yet it doesn’t (always) seem to stick. Moreover, there was the former dean of college admissions at MIT, who appeared to be rather successful at her job until it came out that (you can’t make this stuff up) that she faked her college degrees. I do think, however, that the probability of a fraudulent person doing something real and meaningful in the world is probably considerably less than the infamous 1 out of 10 ratio of success to failure that venture people always bandy about, or at least considerably less than someone who's not a Faker McFakerpants. Plus, as the MIT example shows, there’s always the risk that someone finds out about it, and it leads to a high-profile debacle. Imagine if Elizabeth Holmes said that she actually graduated from Stanford (instead of admitting to dropping out (worn as a badge of honor?)). Would there be any chance she would have taken her scam this far without someone blowing the whistle? Overall, I think there’s a substantial long term risk in funding liars and cheats (duh?).

Another possibility, though, is that venture capitalists will fund people who are liars and cheats because they don’t care about building a viable long term business. All they care about is pumping the business up and selling it off to the next bidder. Perhaps the venture capitalists will invest in a charming con-artist because someone not, ahem, constrained by the details of reality might be a really good salesman. I don’t know, but the cynic in me says that this may be the answer more often than not. One might say, well, whatever, who cares if some Silicon Valley billionaires lose a couple million dollars. Problem is, implicit in this possibility is that somebody is losing out, most likely some other investors along the way. Just as bad, rewarding cheaters erodes everyone’s sense of trust in the system. This is particularly aggravating in cases when the company is couched in moral or ethical terms—and in situations where patient health is involved, everything suddenly becomes that much more serious still.

Overall, one eye-opening aspect of all this for me as an academic is that while we value integrity, skepticism and evidence very highly, business values things like “passion” more than we do. I don’t know that an imposition of academic values would have necessarily caught something like Theranos earlier on and all the other lesser known cases out there, but I would like to think that it would. Why are these values not universal, though? After all, our role in academia is that of evaluation, of setting a bar that employers value. In a way, our student’s aren’t really paying for an education per se—rather, they are paying for our evaluation, which is a credential that will get them a job; in a sense, it’s their future employers that are paying for the degree. Why doesn’t this work when someone fakes a degree? When someone fakes data?

Here’s a thought. One way to counter the strategy of funding fakers and frauds would be for us to make this information public. It would be very difficult, then, to pump up the value of the company with such a cloud hanging over it, and so I think this would be a very effective deterrent. The biggest problem with this plan is the law. Making such information public can lead to big defamation lawsuits directed at the university and perhaps the faculty personally, and I’ve heard of universities losing these lawsuits even if they have documented proof of the fraud. So naturally, universities generally advise faculty against any public declarations of this sort. I don’t know what to do about that. It seems that with the laws set up the way they are, this option is just not viable most of the time.

I think the only real hope is that venture capitalists eventually decide that integrity actually does matter for the bottom line. I certainly don't have any numbers on this, but I know of at least one venture capital firm that claims success rates of 4 in 10 by taking a long view and investing carefully in the success of the people and ventures they fund. I would assume that integrity would matter a lot in that process. And I really do believe that at the end of the day in industry, integrity and reality really do trump hype and salesmanship, just like in academia. I don’t know a lot of CEOs, but one of my heroes is Ron Cook, CEO of Biosearch Technologies, a great scientist, businessman, and a person of integrity. I think it’s not coincidental that Ron has a PhD. For real.

Update in response to comments, 7/22/2016:
Got a comment from Anonymous and Sri saying that this post is overblowing the issue and unfairly impugns the venture capital industry. I would agree that perhaps some elements of this post are a bit overblown, and I certainly have no idea what the extent of this particular issue (knowingly funding fakers) is. This situation probably doesn't come up in the majority of cases, and it may be relatively rare. All that said, I understand that my post is short on specifics and data when it comes to funding known fakers and looking the other way, but I think it will be impossible to get data on this for the very same reason: fear of defamation lawsuits. You just can't say anything specific without being targeted by a defamation suit that you will probably lose even if you have evidence of faking. So where are you going to get this data from?

And it's true that I personally don't have enough anecdotes to consider this data. But I can say that essentially every single person I've discussed this with tells me the same thing: even if you say something, the venture capitalists won't care. In at least some cases, they have specific personal examples.

Also, note that I am not directly calling out the integrity of the venture capitalists themselves, but rather just pointing out that personal integrity of who they fund is not necessarily as big a factor in their decision making as I would have thought. My point is not so much about the integrity of venture capitalists—I suspect they are just optimizing to their objective function, which is return on investment. I just think that it's shady at a societal level that the integrity of who they fund is apparently less important to them than we in academia would hope. Let me ask you this: in your department, would you hire someone on the faculty knowing that they had faked their degree? I'm guessing the answer is no, and for good reason. The question is why those same reasons don't matter when venture capital are deciding who to fund.

Reproducibility, reputation and playing the long game in science

Every so often these days, something will come up again about how most research findings are false. Lots of ink has already been spilled on the topic, so I won’t dwell on the reproducibility issue too long, but the whole thing has gotten me thinking more and more about the meaning and consequences of scientific reputation.

Why reputation? Reputation and reproducibility are somewhat related but clearly distinct concepts. In my field (I guess?) of molecular biology, I think that reputation and reproducibility are particularly strongly correlated because the nature of the field is such that perceived reproducibility is heavily tied to the large number of judgement calls you have make in the course of your research. As such, perhaps reputation has evolved as the best way to measure reproducibility in this area.

I think that this stands in stark contrast with the more common diagnosis one sees these days for the problem of irreproducibility, which is that it's all down to statistical innumeracy. Every so often, I’ll see tweets like this (names removed unless claimed by owner):

The implication here is that the problem with all this “cell” biology is that the Ns are so low as to render the results statistically meaningless. The implicit solution to the problem is then “Isn’t data cheap now? Just get more data! It’s all in the analysis, all we need to do is make that reproducible!” Well, if you think that github accounts, pre-registered studies and iPython notebooks will magically solve the reproducibility problem, think again. Better statistical and analysis management practices are of course good, but the excessive focus on these solutions to me ignores the bigger point, which is that, especially in molecular and cellular biology, good judgement about your data and experiments trumps all. (I do find it worrying that statistics has somehow evolved to the point of absolving ourselves of the responsibility for the scientific inferences we make ("But look at the p-value!"). I think this statistical primacy is perhaps part of an bigger—and in my opinion, ill-considered—attempt to systematize and industrialize scientific reasoning, but that’s another discussion.)

Here’s a good example from the (infamous?) study claiming to show that aspartame induces cancer. (I looked this over a while ago given my recently acquired Coke Zero habit. Don’t judge.) Here’s a table summarizing their results:

The authors claim that this shows an effect of increased lymphomas and leukemias in the female rats through the entire dose range of aspartame. And while I haven’t done the stats myself, looking at the numbers, the claim seems statistically valid. But the whole thing really hinges on the one control datapoint for the female rats, which is (seemingly strangely) low compared to virtually everything else. If that number was, say, 17% instead of 8%, I’m guessing essentially all the statistical significance would go away. Is this junk science? Well, I think so, and the FDA agrees. But I would fully agree that this is a judgement call, and in a vacuum would require further study—in particular, to me, it looks like there is some overall increase in cancers in these rats at very high doses, and while it is not statistically significant in their particular statistical treatment, my feeling is that there is something there, although probably just a non-specific effect arising from the crazy high doses they used.

Hey, you might say, that’s not science! Discarding data points because they “seem off” and pulling out statistically weak “trends” for further analysis? Well, whatever, in my experience, that’s how a lot of real (and reproducible) science gets done.

Now, it would be perfectly reasonable of you to disagree with me. After all, in the absence of further data, my inklings are nothing more than an opinion. And in this case, at least we can argue about the data as it is presented. In most papers in molecular biology, you don’t even get to see the data from all experiments they didn’t report for whatever reason. The selective reporting of experiments sounds terrible, and is probably responsible for at least some amount of junky science, but here’s the thing: I think molecular biology would be uninterpretable without it. So many experiments fail or give weird results for so many different reasons, and reporting them all would leave an endless maze that would be impossible to navigate sensibly. (I think this is a consequence of studying complex systems with relatively imprecise—and largely uncalibrated—experimental tools.) Of course, such a system is ripe for abuse, because anyone can easily leave out a key control that doesn’t go their way under the guise of “the cells looked funny that day”, but then again, there are days where the cells really do look funny. So basically, in the end, you are stuck with trust: you have to trust that the person you’re listening to made the right decisions, that they checked all the boxes that you didn’t even know existed, and that they exhibited sound judgement. How do you know what work to follow up on? In a vacuum, hard to say, but that’s where reputation comes in. And when it comes to reputation, I think there’s value in playing the long game.

Reputation comes in a couple different forms. One is public reputation. This is the one you get from talks you give and the papers you publish, and it can suffer from hype and sloppiness. People do still read papers and listen to talks (well, at least sometimes), and eventually they will notice if you cut corners and oversell your claims. Not much to say about this except that one way to get a good public reputation is to, well, do good science! Another important thing is to just be honest. Own up to the limitations of your work, and I’ve found that people will actually respect you more. It’s pretty easy to sniff out someone who’s being disingenuous (as the lawyerly answers from Elizabeth Holmes have shown), and I think people will actually respect you more if you just straight up say what you really think. Plus, it makes people think you’re smart if you show you’ve already thought about all the various problems.

Far more murky is the large gray zone of private reputation, which encompasses all the trust in the work that you don’t see publicly. This is going out to dinner with a colleague and hearing “Oh yeah, so-and-so is really solid”… or “That person did the same experiment 40 times in grad school to get that one result” or “Oh yeah, well, I don’t believe a single word out of that person’s mouth.” All of which I have heard, and don’t let me forget my personal favorite “Mr. Artifact bogus BS guy”. Are these just meaningless rumors? Sometimes, but mostly not. What has been surprising to me is how much signal there is in this reputational gossip relative to noise—when I hear about someone with a shady reputation, I will often hear very similar things independently from multiple sources.

I think this is (rightly) because most scientists know that spreading science gossip about people is generally something to be done with great care (if at all). Nevertheless, I think it serves a very important purpose, because there’s a lot of reputational information that is just hard to share publicly. Many reasons for this, one of them being that the burden of proof for calling someone out publicly is very high, the potential for negative fallout is large, and you can easily develop your own now-very-public reputation for being a bitter, combative pain in the ass. A world in which all scientists called each other out publicly on everything would probably be non-functional.

Of course, this must all be balanced against the very significant negatives to scientific gossip. It is entirely possible that someone could be unfairly smeared in this way, although honestly, I’m not sure how many instances of this I’ve really seen. (I do know of one case in which one scientist supposedly started a whisper campaign against another scientist about their normalization method or something suitably petty, although I have to say the concerns seemed valid to me.)

So how much gossip should we spread? For me, that completely depends on the context. With close friends, well, that’s part of the fun! :) With other folks, I’m of course far more restrained, and I try to stick to what I know firsthand, although it’s impossible to give a straight up rule given the number of factors to weigh. Are they asking for an evaluation of a potential collaborator? Are we discussing a result that they are planning to follow up on in the lab, thus potentially harming a trainee? Will they even care what I say either way? An interesting special case is trainees in the lab. I think they actually stand to benefit greatly from this informal reputational chatter. Not only do they learn who to avoid, but even just knowing the fact that not everyone in science can be trusted is a valuable lesson.

Which leads to another important problem with private reputations: if they are private, what about all the other people who could benefit from that knowledge but don’t have access to it? This failure can manifest in a variety of ways. For people with less access to the scientific establishment (smaller or poorer countries, e.g.), you basically just have to take the literature at face value. The same can be true even within the scientific establishment; for example, in interdisciplinary work, you’ll often have one community that doesn’t know the gossip of another (lots of examples where I’ll meet someone who talks about a whole bogus subfield without realizing it’s bogus). And sometimes you just don’t get wind in time. The damage in terms of time wasted is real. I remember a time when our group was following up a cool-seeming result that ended up being bogus as far as we could tell, and I met a colleague at a conference, told her about it, and she said they saw the same thing. Now two people know, and perhaps the handful of other people that I’ve mentioned this to. That doesn’t seem right.

At this point, I often wonder about a related issue: do these private reputations even matter? I know plenty of scientists with widely-acknowledged bad reputations who are very successful. Why doesn’t it stick? Part of it is that our review systems for papers and grants just don’t accommodate this sort of information. How do you give a rational-sounding review that says “I just don’t believe this”? Some people do give those sorts of reviews, but come across as, again, bitter and combative, so most don’t. Not sure what to do about this problem. In the specific case of publishing papers, I often wonder why journal editors don’t get wind of these issues. Perhaps they just are in the wrong circles? Or maybe there are unspoken union rules about ratting people out to editors? Or maybe it’s just really hard not to send a paper to review if it looks strong on the face of it, and at that point, it’s really hard for reviewers to do anything about it. It is possible that preprints and more public discussion may help with this? Of course, then people would actually have to read each other’s papers…

That said, while the downsides of a bad private reputation may not materialize as often as we feel they should, the good news is that I think the benefits to a good private reputation can be great. If people think you do good, solid work, I think that people will support you even if you’re not always publishing flashy papers and so forth. It’s a legitimate path to success in science, and don’t let the doom and gloomers and quit-lit types tell you otherwise. How to develop and maintain a good private reputation? Well, I think it’s largely the same as maintaining a good public one: do good science and don’t be a jerk. The main difference is that you have to do these things ALL THE TIME. There is no break. Your trainees and mentors will talk. Your colleagues will talk. It’s what you do on a daily basis that will ensure that they all have good things to say about you.

(Side point… I often hear that “Well, in industry, we are held to a different standard, we need things to actually work, unlike in academia.” Maybe. Another blog post on this soon, but I’m not convinced industry is any better than academia in this regard.)

Anyway, in the end, I think that molecular biology is the sort of field in which scientific reputation will remain an integral part of how we assess our science, for better or for worse. Perhaps we should develop a more public culture of calling people out like in physics, but I’m not sure that would necessarily work very well, and I think the hostile nature of discourse in that field contributes to a lack of diversity. Perhaps the ultimate analysis of whether to spread gossip or do something gossip-worthy is just based on what it takes for you to get a good night’s sleep.

Thoughts on the NEJM editorial: what’s good for the (experimental) goose is good for the (computational) gander

Huge Twitter explosion about this editorial in the NEJM about “research parasites”. Basically, the authors say that computational people interested in working with someone else’s data should work together with the experimenters (which, incidentally, is how I would approach something like that in most cases). Things get a bit darker (and perhaps more revealing) when they also call out “research parasites”–aka “Mountain Dew chugging computational types”, to paraphrase what I’ve heard elsewhere–who are are to them just people sitting around, umm, chugging Mountain Dew while banging on their computers, stealing papers from those who worked so hard to generate these datasets.

So this NEJM editorial is certainly wrong on many counts, and I think that most people have that covered. Not only that, but it is particularly tone-deaf: “… or even use the data to try to disprove what the original investigators had posited.” Seriously?!?

The response has been particularly strong from the computational genomics community, who are often reliant on other people’s data. Ewan Birney had a nice set of Tweets on the topic, first noting that “For me this is the start of clinical research transitioning from a data limited to an analysis limited world.”, noting further that “This is what mol. biology / genomics went through in the 90s/00s and it’s scary for the people who base their science on control of data.” True, perhaps.

He then goes on to say: “1. Publication means... publication, including the data. No ifs, no buts. Patient data via restricted access (bonafide researcher) terms.”

Agreed, who can argue with that! But let’s put this chain of reasoning together. If we are moving to an “analysis limited world”, then it is the analyses that are the precious resource. And all the arguments for sharing data are just as applicable to sharing analyses, no? Isn’t the progress of science impeded by people not sharing their analyses? This is not just an abstract argument: for example, we have been doing some ATAC-seq experiments in the lab, and we had a very hard time finding out exactly how to analyze that data, because there was no code out there for how to do it, even in published papers (for the record, Will Greenleaf has been very kind and helpful via personal communication, and this has been fine for us).

What does, say, Genome Research have to say about it? Well, here’s what they say about data:

Genome Research will not publish manuscripts where data used and/or reported in the paper is not freely available in either a public database or on the Genome Research website. There are no exceptions.

Uh, so that’s pretty explicit. And here’s what they say about code:

Authors submitting papers that describe or present a new computer program or algorithm or papers where in-house software is necessary to reproduce the work should be prepared to make a downloadable program freely available. We encourage authors to also make the source code available.

Okay, so only if there’s some novel analysis, and then only if you want to or if someone asks you. Probably via e-mail. To which someone may or may not respond. Hmm, kettle, the pot is calling…

So what happens in practice at Genome Research? I took a quick look at the first three papers from the current TOC (1, 2, 3).

The first paper has a “Supplemental PERL.zip” that contains some very poorly documented code in a few files and as far as I can tell, is missing a file called “mcmctree_copy.ctl” that I’m guessing is pretty important to the running the mcmctree algorithm.

The third paper is perhaps the best, with a link to a software package that seems fairly well put together. But still, no link to the actual code to make the actual figures in the paper, as far as I can see, just “DaPars analysis was performed as described in the original paper (Masamha et al. 2014) by using the code available at https://code.google.com/p/dapars with default settings.”

The second paper has no code at all. They have a fairly detailed description of their analysis in the supplement, but again, no actual code I could run.

Aren’t these the same things we’ve been complaining about in experimental materials and methods forever? First paper: missing steps of a protocol? Second paper: vague prescription referencing previous paper and a “kit”? Third paper: just a description of how they did it, just like, you know, most “old fashioned” materials and methods from experimental biology papers.

Look, trust me, I understand completely why this is the case in these papers, and I’m not trying to call these authors out. All I’m saying is that if you’re going to get on your high horse and say that data is part of the paper and must be distributed, no ifs, no buts, well, then distribute the analyses as well–and I don’t want to hear any ifs or buts. If we require authors to deposit their sequence data, then surely we can require that they upload their code. Where is the mandate for depositing code on the journal website?

Of course, in the real world, there are legitimate ifs and buts. Let me anticipate one: “Our analyses are so heterogeneous, and it’s so complicated for us to share the code in a usable way.” I’m actually very sympathetic to that. Indeed, we have lots of data that is very heterogeneous and hard to share reasonably–for anyone who really believes all data MUST be accessible, well, I’ve got around 12TB of images for our next paper submission that I would love for you to pay to host… and that probably nobody will ever use. Not all science is genomics, and what works in one place won’t necessarily make sense elsewhere. (As an aside, in computational applied math, many people keep their codes secret to avoid “research parasites”, so it’s not just data gatherers who feel threatened.)

Where, might you ask, is the moral indignation on the part of our experimental colleagues complaining about how computational folks don’t make their codes accessible? First off, I think many of these folks are in fact annoyed (I am, for instance), but are much less likely to be on Twitter and the like. Secondly, I think that many non-computational folks are brow-beaten by p-value toting computational people telling them they don’t even know how to analyze their own data, leading them to feel like they are somehow unable to contribute meaningfully in the first place.

So my point is, sure, data should be available, but let’s not all be so self-righteous about it. Anyway, there, I said it. Peace. :)

PS: Just in case you were wondering, we make all our software and processed data available, and our most recent paper has all the scripts to make all the figures–and we’ll keep doing that moving forward. I think it's good practice, my point is that reasonable people could disagree.

Update: Nice discussion with Casey Bergman in the comments.
Update (4/28/2016): Fixed links to Genome Research papers (thanks to Quaid Morris for pointing this out). Also, Quaid pointed out that I was being unreasonable, and that 2/3 actually did provide code. So I looked at the next 3 papers from that issue (4, 5, 6). Of these, none of them had any code provided. For what it's worth, I agree with Quaid that it is not necessarily reasonable to require code. My point is that we should be reasonable about data as well.

Is the academic work ethic really toxic?

Every so often, I’ll read something or other about how the culture of work in academia is toxic, encouraging people to work 24/7/52 (why do people say 24/7/365?) and thus ignore all other aspects of their existence and in the process destroying their life. As I’ve written before, I think this argument gets it backwards. I think most academics work hard because they want to and are immersed in what they are doing, not because of the “culture”. It is the conflation of hours and passion that lead to confusion.

Look, I know people who are more successful than I am and work less than I do. Good for them! That doesn’t mean I’m going to start working less hard. To me, if you’re thinking “I need to work X hours to get job Y/award Z”, well, then you’re in the wrong line of work. If you’re thinking “I really need to know about X because, uh, I just need to know” then academia might be for you. Sure, sometimes figuring out X requires a lot of work, and there is a fair amount of drudgery and discipline required to turn an idea into a finished paper. Most academics I know will make the choice to do that work. Some will do it at a pace I would find unmanageable. Some will do it at a pace I find lethargic. I don’t think it really matters. I read a little while ago that Feng Zhang goes back to work every day after dinner and works until 3am doing experiments himself in the lab (!). I couldn’t do that. But again, reading about Zhang, I think it’s pretty clear that he does it because he has a passion for his work. What’s wrong with that? If he wants to work that way, I don’t see any reason he should be criticized for it. Nor, conversely, lionized for it. I think we can praise his passion, though. Along those lines, I know many academics who are passionate about their work and thus very successful, all while working fairly regular hours (probably not 40/week, but definitely not 80/week), together with long vacations. Again, the only requirement for success in science is a desire to do it, along with the talent and dedication to finish what you start.

I think this conflation of hours and passion leads to some issues when working with trainees. To me, I most enjoy working with people who have a passion for their work. Often, but not always, this means that they work long-ish hours. If someone is not motivated, then a symptom is sometimes working shorter hours–or, other times, working long hours but not getting as much done. If we’re to the point where I’m counting someone’s hours, though, then it’s already too late. For trainees, if your PI is explicitly counting hours, then that means either you should find a new PI or carefully consider why your PI is counting your hours. What’s important is that both parties should realize that hours are the symptom, not the underlying condition.

Bragging about data volume is lame

I've noticed a trend in some papers these days of bragging about the volume of data you collect. Here's an example (slightly modified) from a paper I was just looking at "We analyzed a total of 293,112 images." Often times, these numbers serve no real purpose except to highlight that you took a lot of data, which I think is sort of lame.

Of course, numbers in general are good and are an important element in describing experiments. Like "We took pictures of at least 5000 cells in 592 conditions." That gives a sense of the scale of the experiment and is important for the interpretation. But if you just say "We imaged a total of 2,948,378 cells", then that provides very little useful information about why you imaged all those cells. Are they all the same? Is that across multiple conditions? What is the point of this number except to impress?

And before you leave a comment, yes, I know we did that in this paper. Oops. I feel icky.