Just wrote a post on how to review a paper, but I realized that there's another step to review, which is how to deal with revisions. Anyway, here's a small follow-up on how to re-review a paper, which is when the authors revise the manuscript and resubmit for you to look at and approve.
1. If you now find something problematic from the original submission that made it into the revision, well, too bad. In my opinion, you are not allowed two bites at the apple. I realize that this is sort of "bad" in the sense that the manuscript may be published with a flaw, and reasonable people could disagree on this point. But this is my policy, because it's just absolutely deflating to deal with this as an author. I should say that it is pretty rare for something really critical to only surface at the second round. When I've gotten these second round criticisms, they are typically just for inconsequential stuff that the reviewer is trying to trump up because they are a jerk.
2. Keep it short. Like, two lines or so.
3. Don't get mad if they didn't do everything you asked. I had one reviewer for a paper who clearly didn't like our work ask us repeatedly for more experiments and by the end basically just said something to the effect of "I just feel like they should have done an experiment." Not cool.
At this stage, the editors have probably basically signed off on it (though not always, we all know horror stories), so I don't think it's worth getting too involved.
Another, somewhat related issue: how to review a paper that you've already reviewed for another journal. Only been in that situation a few times myself, and it's always weird. I guess you can just recycle your previous review, unless the paper has changed somewhat. Or just decline to review the paper (seen that as well). Better thing is to just try and get the paper accepted at the first journal so that you don't ever have to see it again.
Saturday, April 26, 2014
Friday, April 25, 2014
“Why don’t you just use [some old boring tech] instead?”
One common criticism/comment about measurement techniques that I hear scientists hurl at each other is “Why don’t you just use [some old boring tech] to measure that instead?” Like: “Couldn’t you just use RT-qPCR instead of RNA FISH?” Or “Couldn’t you get the same results with a western blot?” Or “Do you really need single cell analysis to get that result?” These points are fair ones, and often it is completely true you could use something less precise, cheaper, more standard, etc. to measure something. I sometimes say (or at least think) these sort of things myself.
But I think this ignores the fact that once a new, high resolution and more precise technique becomes standard, it can free your mind to think in new ways. Take an example from numerical methods (I forget who told me this example–I want to say it was my office mate at Courant, the absolutely brilliant Yoichiro Mori). When I type in sin(1.4787) into MATLAB, it just gives me the answer. It gives me the answer out to far more decimal places than probably need, but whatever, it just works. The most important part is that *if* what I’m doing requires more decimal places, I don’t need to think “oh, I should probably use this other algorithm for that, maybe that’s why this thing isn’t working”–no, it just works all the way, every time, whether I need it or not. Not having to worry about it frees your mind from the details and let’s you ask things you couldn’t or wouldn’t ask otherwise.
For a biological example, let’s say I need to measure a fold change in transcript abundance. In our lab, we’d probably do RNA FISH, which gives us absolute numbers and single cell counts, etc. Now, someone could say, well, do you need counts in single cells? Do you need absolute numbers? Why not just do RT-qPCR? Sure, fair enough. But to the extent that we believe RNA FISH is more accurate, then why not? Then I just don’t have to worry about all the controls, etc. And if I want to ask a question of my data that does require more accurate numbers, well, then I have it. The point is that I have the answer and I can move on to the next thing.
But I think this ignores the fact that once a new, high resolution and more precise technique becomes standard, it can free your mind to think in new ways. Take an example from numerical methods (I forget who told me this example–I want to say it was my office mate at Courant, the absolutely brilliant Yoichiro Mori). When I type in sin(1.4787) into MATLAB, it just gives me the answer. It gives me the answer out to far more decimal places than probably need, but whatever, it just works. The most important part is that *if* what I’m doing requires more decimal places, I don’t need to think “oh, I should probably use this other algorithm for that, maybe that’s why this thing isn’t working”–no, it just works all the way, every time, whether I need it or not. Not having to worry about it frees your mind from the details and let’s you ask things you couldn’t or wouldn’t ask otherwise.
For a biological example, let’s say I need to measure a fold change in transcript abundance. In our lab, we’d probably do RNA FISH, which gives us absolute numbers and single cell counts, etc. Now, someone could say, well, do you need counts in single cells? Do you need absolute numbers? Why not just do RT-qPCR? Sure, fair enough. But to the extent that we believe RNA FISH is more accurate, then why not? Then I just don’t have to worry about all the controls, etc. And if I want to ask a question of my data that does require more accurate numbers, well, then I have it. The point is that I have the answer and I can move on to the next thing.
(Cost is of course an important consideration as well, and one that I'm not considering here. Costs change, though, and I also think that scientists often don't factor in the time required. RT-PCR is relatively cheap, but many fail to take into consideration the cost of the time used to validate primers, build standard curves, etc.)
I think this “arbitrary precision” principle is important for building assays that we can build upon. I think that it is generally true that it is hard to build new assays out of parts that are finicky and require a lot of careful consideration. Take Sanger sequencing. When we sequence, we don’t usually have to worry too much about how the loaded the sample, etc. It just works and we take it as such. Same with Illumina sequencing, for the most part. Or buying an off-the-shelf microscope. Yes, these methods and tools have important tricky points that we have to watch out for, but the point is that you don’t have to think about them most of the time. It makes it just that much easier to innovate further upon them. It's much harder to build something on top of a coin flip.
Anyway, just something to think about when someone introduces a fancy new method to measure something. Going to try and stay more open minded myself.
I think this “arbitrary precision” principle is important for building assays that we can build upon. I think that it is generally true that it is hard to build new assays out of parts that are finicky and require a lot of careful consideration. Take Sanger sequencing. When we sequence, we don’t usually have to worry too much about how the loaded the sample, etc. It just works and we take it as such. Same with Illumina sequencing, for the most part. Or buying an off-the-shelf microscope. Yes, these methods and tools have important tricky points that we have to watch out for, but the point is that you don’t have to think about them most of the time. It makes it just that much easier to innovate further upon them. It's much harder to build something on top of a coin flip.
Anyway, just something to think about when someone introduces a fancy new method to measure something. Going to try and stay more open minded myself.
Sitting in the very front of the plane is no fun
I've never sat at the front of the plane before–my few experiences with plane boundary conditions have been at the other end of things. But yesterday I got on a really small plane and there was a seat in front open (no, definitely not first class) and they directed me there. It was awful! You can't do anything bad like play with your laptop at the wrong time, etc. because they are hovering over you like a hawk. Wish I could have gone back to sit in seat 3F again...
Sunday, April 20, 2014
Talking with my mom about GMOs
(I normally steer well clear of anything remotely political on the blog for obvious reasons. But this is not really about politics. Sort of. Whatever.)
I just had a huge argument with my mom about genetically modified organisms (GMOs). My mom is staunchly anti-GMO, and will not change her position no matter what I say. Despite the argument that the scientific consensus is that genetically modified organisms are not intrinsically a bad thing (indeed, essentially all scientists I have met who are even tangentially qualified to speak on the topic agree), my mom simply will not budge. My mom belongs in some region of a Venn diagram of people who in the most extreme intersection are simultaneously reasonably (often formidably) well-educated, believe in global warming, are anti-vaccination, and are anti-GMO. They are also probably very likely to eat gluten-free diets and kale chips. (Note that my mom is neither anti-vaccine nor gluten-free. I have not asked her about kale chips. For the record, my own personal political views are that I am neutral on kale chips.)
What's interesting here is that if you push these folks on climate change, they will probably tell you that the scientific consensus is in very strong agreement that anthropogenic climate change is real. Why would the same argument based on science not apply to genetically modified organisms or vaccines? I think that reveals a more fundamental truth: nominally, you may expect folks like this, who are well-educated and most likely politically labeled as liberal, to be intrinsically pro-science, and perhaps that is true on some level. But I think that a more accurate characterization would be "pro-nature" or "pro-environment" or maybe "anti-man-made". If this coincides with science, then science is right. But if not, well, science is wrong. The mentality is not much different than those on the other end of the political spectrum, just with a different set of beliefs.
(Again, for the record, I am both pro-Nature and pro-Science. I would happily publish papers at either one.) (Haha, bad joke!) (You know you love it.)
I think we scientists had better keep this reality in mind. Just remember that virtually nobody outside of science really knows what the hell we are talking about. Some people may want to support us based on some alignment of their belief system and priorities with some aspects of our beliefs and priorities. Fine. But there are very few people out there who support us based on an actual, real understanding of what we do. I'm not saying this is good or bad, rather that it's a reality and we should brace ourselves accordingly for when those belief systems shift. Overall, perhaps we're lucky to be just a rounding error in the federal budget. I think this reflects the fact that we are a rounding error in most people's minds.
We can of course rightly argue that we provide society with incredible (and outsized) benefits in that scientific progress has led to an enormous gains in virtually every aspect of human life. So perhaps the fact that there are people out there whose interests align with ours, however imperfectly the motivations may match up, is good. But I think that we have to be very careful about relying on a system that is set up in such a way. Here's a Feynman quote that I serendipitously happened across just now that is particularly apropos:
I would like to add something that’s not essential to the science, but something I kind of believe, which is that you should not fool the layman when you’re talking as a scientist. I am not trying to tell you what to do about cheating on your wife, or fooling your girlfriend, or something like that, when you’re not trying to be a scientist, but just trying to be an ordinary human being. We’ll leave those problems up to you and your rabbi. I’m talking about a specific, extra type of integrity that is not lying, but bending over backwards to show how you are maybe wrong, that you ought to have when acting as a scientist. And this is our responsibility as scientists, certainly to other scientists, and I think to laymen.
For example, I was a little surprised when I was talking to a friend who was going to go on the radio. He does work on cosmology and astronomy, and he wondered how he would explain what the applications of this work were. “Well,” I said, “there aren’t any.” He said, “Yes, but then we won’t get support for more research of this kind.” I think that’s kind of dishonest. If you’re representing yourself as a scientist, then you should explain to the layman what you’re doing–and if they don’t want to support you under those circumstances, then that’s their decision.
(I actually came across this quote in this blog post about the recent, umm, "back and forth" between Lior Pachter and Manolis Kellis. Scientific celebrity death match: Fight!)
I guess I think that it's a sign of a highly civilized society that we can have people sit around and spend precious resources thinking about stuff that quite often just doesn't matter. Would that be enough justification for the rest of society? Should I be okay with my mom supporting science funding despite her views? Is it possible or even desirable to live a life that is completely free of hypocrisy? To the latter question, I think the answer is no. People would be so boring otherwise.
Saturday, April 19, 2014
How to review a paper
Or, I should say, how I review a paper.
Peer review is a mess. We all know it, and have written about it endlessly, including myself. And we’ve also railed against a system in which we do all the work for the benefit of the publishers. But wait: if we are doing all the work, then we should be able to bend this system to our collective will, right? When we complain about bad reviews, just remember that we ourselves are the ones giving these terrible reviews. So our goal should be to give good reviews!
How do you do that? Here are some principles I try to think about and follow:
1. Don’t review papers you think will be bad. You can usually tell if a paper is low quality (and thus unlikely to make the cut) just by looking at the title and abstract. If you think it’s a waste of your energy, it probably is, and so don’t waste your time reviewing it. Somebody else will do it. Or not. It’s not your obligation to do so. Beware also the ego issue.
2. Give the authors the benefit of the doubt. The authors have been thinking about the problem for years, you have been thinking about it for a couple hours. So if you find a particular point confusing, either try to figure it out and really understand it, or just give the authors the benefit of the doubt that they did the right thing. They may not have, but so what? The alternative is worse, in my opinion. I hate getting the “The authors must check for XYZ if they want to make that claim”, when in fact we already did exactly that.
3. Stick to the evidence and the claims. I stay away from any of that crap about “impact". To me, that is an editorial responsibility. I try my best to just evaluate the evidence for the claims that the authors are making. If the authors claim something that they can’t based on their evidence, then I just say that the authors should alter their claim. And I try to say what they can claim, rather than just what they can’t claim. I generally do NOT propose new experiments to support their claim.
4. Do not EVER expand the scope of the paper. It is not your paper, and so it is neither your responsibility nor mandate to dictate the authors’ research priorities. The worst reviews are the ones in which the reviewers ask the authors to do a whole new PhD’s worth of studies. Often, these types of comments include vague prescriptions about novelty and mechanism. To wit, check out this little snippet from another reviewer for a paper I reviewed recently: "However, the major pitfall of current study is not to provide any novel information/mechanism behind the heterogeneity.” What is the author supposed to do with that?
5. Worth repeating–do not propose new experiments. Those reviews with a laundry list of experiments are very disheartening, and usually don’t add much to the paper. Remember that experiments are expensive, and sometimes the main author has left the lab and it’s very hard to even do the suggested experiments. Again, far better to just have them alter their claims in line with what their available data shows. I will sometimes explicitly say “The authors do not need to do an experiment” just to make that clear to the authors and the editor.
6. Be specific. If something is unclear, say exactly what it is that was confusing. If some claim is too broad, then give a specific alternative that the authors should consider (I’ll say “the authors could consider and discuss alternative hypothesis XYZ.”). This helps authors know exactly what they should do to make the paper go through. Also clearly delineate what are critical points and what are minor points.
7. Be positive and nice. Every paper represents a lot of someone’s blood, sweat and tears, usually a young scientist, and scathing reviews can be personally devastating. If you have to be negative (which is difficult, following the above rubric), then try not to phrase your review in terms like “I don’t know why anybody would ever do this”. Here’s an example of the opening lines from a review we got a while ago: "My opinion is that this manuscript is not very well thought through and of rather low quality. The authors' misconceptions are most obvious in their gross misstatement… [some relatively minor and inconsequential misstatement]”. Ouch. What’s the point of that? That review ended with “If [after doing a bunch of proposed experiments that don’t make sense] they find that [bunch of stuff that is irrelevant] they would begin to address their question.” It’s very belittling to basically imply that after all this work, we are still at square one. Not true, not positive and not nice.
8. (related) Write as though you were not anonymous. That will help with all the above issues.
One other note: I realize that for editors, even academic editors, the issues of novelty and impact are difficult to gauge and that they feel the need to lean on the reviewers for this information. Fine, I get that. But I will not provide it. Sorry.
Anyway, my main point is that WE run this system. It is within our power to change it for the better. Train your people and yourself to be better reviewers, and maybe we will all even be better people for it.
Peer review is a mess. We all know it, and have written about it endlessly, including myself. And we’ve also railed against a system in which we do all the work for the benefit of the publishers. But wait: if we are doing all the work, then we should be able to bend this system to our collective will, right? When we complain about bad reviews, just remember that we ourselves are the ones giving these terrible reviews. So our goal should be to give good reviews!
How do you do that? Here are some principles I try to think about and follow:
1. Don’t review papers you think will be bad. You can usually tell if a paper is low quality (and thus unlikely to make the cut) just by looking at the title and abstract. If you think it’s a waste of your energy, it probably is, and so don’t waste your time reviewing it. Somebody else will do it. Or not. It’s not your obligation to do so. Beware also the ego issue.
2. Give the authors the benefit of the doubt. The authors have been thinking about the problem for years, you have been thinking about it for a couple hours. So if you find a particular point confusing, either try to figure it out and really understand it, or just give the authors the benefit of the doubt that they did the right thing. They may not have, but so what? The alternative is worse, in my opinion. I hate getting the “The authors must check for XYZ if they want to make that claim”, when in fact we already did exactly that.
3. Stick to the evidence and the claims. I stay away from any of that crap about “impact". To me, that is an editorial responsibility. I try my best to just evaluate the evidence for the claims that the authors are making. If the authors claim something that they can’t based on their evidence, then I just say that the authors should alter their claim. And I try to say what they can claim, rather than just what they can’t claim. I generally do NOT propose new experiments to support their claim.
4. Do not EVER expand the scope of the paper. It is not your paper, and so it is neither your responsibility nor mandate to dictate the authors’ research priorities. The worst reviews are the ones in which the reviewers ask the authors to do a whole new PhD’s worth of studies. Often, these types of comments include vague prescriptions about novelty and mechanism. To wit, check out this little snippet from another reviewer for a paper I reviewed recently: "However, the major pitfall of current study is not to provide any novel information/mechanism behind the heterogeneity.” What is the author supposed to do with that?
5. Worth repeating–do not propose new experiments. Those reviews with a laundry list of experiments are very disheartening, and usually don’t add much to the paper. Remember that experiments are expensive, and sometimes the main author has left the lab and it’s very hard to even do the suggested experiments. Again, far better to just have them alter their claims in line with what their available data shows. I will sometimes explicitly say “The authors do not need to do an experiment” just to make that clear to the authors and the editor.
6. Be specific. If something is unclear, say exactly what it is that was confusing. If some claim is too broad, then give a specific alternative that the authors should consider (I’ll say “the authors could consider and discuss alternative hypothesis XYZ.”). This helps authors know exactly what they should do to make the paper go through. Also clearly delineate what are critical points and what are minor points.
7. Be positive and nice. Every paper represents a lot of someone’s blood, sweat and tears, usually a young scientist, and scathing reviews can be personally devastating. If you have to be negative (which is difficult, following the above rubric), then try not to phrase your review in terms like “I don’t know why anybody would ever do this”. Here’s an example of the opening lines from a review we got a while ago: "My opinion is that this manuscript is not very well thought through and of rather low quality. The authors' misconceptions are most obvious in their gross misstatement… [some relatively minor and inconsequential misstatement]”. Ouch. What’s the point of that? That review ended with “If [after doing a bunch of proposed experiments that don’t make sense] they find that [bunch of stuff that is irrelevant] they would begin to address their question.” It’s very belittling to basically imply that after all this work, we are still at square one. Not true, not positive and not nice.
8. (related) Write as though you were not anonymous. That will help with all the above issues.
One other note: I realize that for editors, even academic editors, the issues of novelty and impact are difficult to gauge and that they feel the need to lean on the reviewers for this information. Fine, I get that. But I will not provide it. Sorry.
Anyway, my main point is that WE run this system. It is within our power to change it for the better. Train your people and yourself to be better reviewers, and maybe we will all even be better people for it.
Wednesday, April 16, 2014
Machine learning, take 2
As mentioned earlier, one of my favorite Gautham quotes is "Would Newton have discovered gravitation by machine learning?" I think the point is solid, that a bunch of data + statistics is not science.
At least not yet. Technically, Newton's brain was a machine, and it came up with gravitation. So it is formally possible to have a machine come up with a theory. And I don't think this argument is just based on a technicality. I was chatting with Gautham yesterday about what a theory is, and doesn't it start with observing a pattern of some kind? Newton had access to centuries (millennia?) of star charts–people had misinterpreted them into epicycles, but the data were there for him. In response to my previous post on statistics, Shankar Mukherji mentioned the work of Hod Lipson, in which they are able to deduce physical laws from the data. Very cool. It seems that progress towards this goal is already underway. My guess is that as we make more progress on machine learning (my completely uninformed bet is on neural network approaches), computers will start to see more seemingly incredible inferences about the world. My other guess is that this will happen a lot sooner than we think.
In the meantime, though, I still think we are pretty far from having Newton in silico, and I think that Gautham's point about real learning vs. (the current state of) machine learning is still a valid one. Until this future of intelligent machines arrives, I think most fields of science will still require a lot more thinking to make sense of the data, and simple classifiers may not yield what we consider scientific insight.
Monday, April 14, 2014
Papers are a lot of work, and some of it is even worth the effort
I often say that the current model for publishing is a complete waste of time, and I still think that's true for so many parts of the publishing process, like dealing with reviews, etc. It's hard for young faculty and even harder for trainees, for whom so much rides on the seemingly arbitrary whims of reviewers and editors. Wouldn't it just be better to post on a blog, I often wonder?
I think deep down I know the answer is no. Not that publishing in a particular journal is really important. But there is something to putting together a well-constructed, high quality paper that is a worthy use of time. Often it feels like finishing a paper is just a bunch of i dotting and t crossing. Yet I've often found that it's in those final stages that we make the most crucial insights. Hedia's lincRNA paper is a good example: it was only towards the end when we were writing it up that we figured out what was really going on with the siRNA vs. ASO oligonucleotide treatment. The details aren't so important, but the point is that this was in some ways the most important finding of the paper, and it was lurking within our data almost until the very end.
I've found the last few weeks before submission to be a stressful period, when you really want to get the paper out the door and at the same time you feel like you're putting a lot on the line that you want to get right. It's exciting but scary to put something out there. And it's especially scary to look at your data again, here at the end of the road, and wonder what it all means after years of hard work. But I feel like this mental incubation period is a necessary part of doing good science, and where many new ideas are born.
Subscribe to:
Posts (Atom)