Let's review I

Good morning, bonjour, buongiorno, guten morgen, dzien dobry, maj po! (Sorry if I left out your language, but I did get you if you are Klingon. In any case, hi there!). It is a bea-u-ti-ful spring day here, and I'm sitting in the sun. And I am in a really good mood, because we just submitted a paper I am very happy about, and I just know that the editors and reviewers are going to love it.Yes, I'm delusional. If it is reviewed at all, it will undoubtedly come back with a long list of deficiencies, concerns, grumbles, and demands for months (or years) of more work. Our wonderful, novel, surprising, and insightful studies will be reduced to what rhymes with ‘it’ and is found on the bottom of bird cages (grit, of course).Less than one percent of the papers I have had the privilege of publishing in my career have actually been accepted in the first review. Although this is a small number, it is clear that this in no way correlated with (1) how solid or important I thought the paper was, (2) the journal to which we had submitted, or (3) how much I loved it. Some papers we submit are accepted after a relatively brief period of revision, whereas others take years to revise and recirculate among different journals (and some of these are still in limbo today).But for now, let me have my lovely fantasy that the paper we just submitted will be loved and admired by editors and reviewers alike, who will praise the many years of innovative work that went into it and offer only minor suggestions for its improvement. I know, ridiculous. But it makes me happy to pretend.So yes, once again, let's talk about reviewing papers. In particular, about why it has become so very difficult to publish even our very best work, why reviewers seem to have become more negative, despite being authors themselves, and finally, what we can do about it. I know I have talked about this before (e.g. Trouble in Flatland II), but I have a few more ideas about this that I think we should discuss.When we are training as graduate students, we are taught how to tear papers apart. This is important, as we learn to be critical and not simply accept whatever authors conclude. We also learn alternative approaches to a problem, technical difficulties, proper statistical analysis, and to evaluate how data support (or not) a conclusion. All very good to know. And we generally do this with papers that have been published. More, we usually do this with papers that have a demonstrated value to the community, because reading such papers in great detail is also a learning exercise. Basically, we are taught critical reading skills.But we are never formally taught how to determine if a paper that has not been published is ready to disseminate to the community at large. One answer to this is that mentors will often give a paper that they have agreed to review to a trainee, with the intention that they will both carefully go over it and compare notes, following which the heavily modified review will be sent in. Many of my colleagues tell me that they do this. But do they really? Do they themselves review the paper, or do they give the trainee's review a cursory look and then, because they're pushed for time, send it off when they realize they have missed the deadline for review? Is this fair practice, given that the journal has asked you (the expert) and not your trainee for an evaluation? Or if you are the trainee, are you remembering to put yourself in the position of the authors, or are you aiming to impress your mentor with your critical skills (which, since this is purportedly a ‘training exercise’, you may do over-zealously because you assume you'll be given feedback)?I suggest that it is rare that the mentor who is using this as a learning experience will then discard the trainee's work as ‘much too critical’. More often, I think they point out some other concerns that have been missed. The result is a very lengthy, very critical review with many (usually a ‘boatload’) of demands for further, extensive, and expensive experimentation (and a detailed identification of typos). The mentor might have suggested several weeks-worth of additional work, which becomes months or years. Multiply this by two or three (sometimes more) reviewers (the ones who actually agreed to review) and the result is a very long list.Personally, I have never met a trainee (and meet very few scientists) who can read a manuscript and say, “Hey, I think that the authors have done an amazing, convincing job here and the work should be published as is.” I totally understand that. It would be embarrassing to make such a statement and be told that I had missed many problems and issues with the paper. It is better to air on the negative side than to be marked as ‘un-critical.’ (And before you flame me, I'm not saying that such people do not exist, I only said that I have never met these exceptional trainees).But I do not think that this really explains why reviews have become ever more negative. One contributing factor might be how we tend to review papers. Often, authors will draw a conclusion based on several different approaches, none of which, by themselves, would definitively allow the conclusion, which is reached in assessing all the data (not only the results from one approach). But when we review a paper, we might tend to stop at the point that results from one approach have been shown, and point out the deficiencies in that approach. Then we move on to read the results of another approach, and again stop and note deficiencies. We might even admit that the multiple approaches do support the conclusion, but the individual deficiencies remain (and ‘must be addressed’). And because papers these days are rather long, with lots of data, this list of deficiencies is similarly long. It is unusual, I suggest, that we read a paper and think that, in total, the arguments are sufficiently strong to reach the authors' conclusion, even when we believe that they have it right.Another contributing factor is that the best papers often span across fields, requiring reviewers with different expertise. What might be a completely acceptable standard in one field might be viewed as insufficient in another. Let's say there is a paper on how flu virus infection influences groundhog behavior. The flu expert might feel that using only one or two viruses is insufficient for the study. The behavior expert might want many more behavioral tests. And the groundhog expert might strongly feel that any such study has no value if it was not performed throughout the year, with multiple replicates in succeeding years. And the editor will want to know what happens on Groundhog Day, and how viral infection might affect the time until spring arrives. I might not get to read this paper for years, despite my profound interest (since Dr. Groundhog is a close friend and just went through a bad bout of the flu).Yet another factor may be a social one. For whatever reason, many of us feel powerless against invisible forces that appear to constrain our lives, and for we scientists, this includes the gauntlet of anonymous peer review. So when the power falls to us, we (and by ‘we’ I mean ‘some of us’) unleash our pent up frustrations. This is not idle speculation, this explanation for what seems to be widespread grumpiness in the peer review process. Peer review can be thought of as a zero sum game: (a) if we all accept competent manuscripts we review, we all win (our papers are accepted, happy dances all around); (b) if I accept papers but mine are all rejected, others win but I lose (I read your delighted posts while I rail against the system); (c) we all give very harsh critiques and reject most papers (I don't win, but neither does anyone else). The strategy that gets the most points in competitive, repeated zero sum games is ‘tit-for-tat’: I cooperate until I am betrayed, after which I mirror the response of my previous opponent. In other words, I submit harsh reviews because those are what I get on my own papers. (This takes us into the terrain of identifying reviewers; we'll talk about another time.)Is it valid to critique a paper because the authors did not use a more expensive technique that I favor? Is it okay to demand experiments that I think would be ‘interesting’? Is it appropriate to weigh what we perceive as the ‘value’ of the journal in our decisions of how much additional work should be required?Do we ever ask, “is this paper ready to be read by others?” Yes, we have the authority, since we were asked to be the reviewer (perhaps after many others declined), and yes, how we review a paper is essentially up to us, but do we consider this: if I received this review, would I view it as fair?So, I have a proposal. (Of course I do, I'm Mole.) Most students are given the experience of critically discussing a paper (for example, in journal clubs), but this should not be confused with reviewing. Yes, it is important to learn how to be careful and critical in our reading, but a reviewer's job is to assist the authors in ensuring that the presented data support the conclusions, not to tear the paper apart. We all need to do more in our institutions to help trainees with reviewing papers. For example, there could be a workshop in which participants are grouped according to interest area, and each meeting (maybe each week?), participants in a group will select an unpublished preprint in their chosen area and present their review to the group. There are preprint servers online (such as BioPrescription4; I think that is what Rxiv means, right?) that are great sources for such preprints and are openly available. And each week, we can discuss the extent to which the presented review was appropriate, fair, and one that we would welcome if we were the authors. We critique the critiques. And those who consistently perform well in such a workshop, we would recommend to journals as reviewers, knowing that they will be careful, critical, and knowledgeable, but also even-handed in their evaluations.There also platforms (such as preLights) that encourage the open reviewing of preprints by the community, and trainees can use them as a way to practice and refine their reviewing skills. And given that lots of journals now publish the peer review reports of an article alongside the published version, we could get groups of trainees to analyze these under our supervision. But here's the thing: to do this, those of us who regularly review for journals would have to come to some sort of consensus on what constitutes a helpful review rather than what amounts to a demand for a new research program.We all know that there is a problem with the review process, and despite being authors ourselves, we persist in destroying the hard work of our colleagues in our fields. Let's step back and take a hard look at the review process. Just complaining about it isn't getting us anywhere, is it?Stay with me, I'm sure we'll have a lot more to talk about next time. Meanwhile, I'm going to get some ‘tea’. Oh, and for you Vulcans, ‘Kalifi’!