A Step Toward Fixing Peer Reviews: Sign Them

September 27, 2011
By | 7 Comments

It is common to hear laments about the quality of peer review, the problems with the system, the lack of quality control and the capriciousness of reviewers. Like any author, I have my war stories, though the real war stories more resemble the incredible revelations by Carole Blair, Julie Brown and Leslie Baxter in their classic essay “Disciplining the Feminine,” where they chronicle feminist-bating in what is clearly a corrupt review process. Countless other authors have made similar points about the politics of peer review (see Blair, Brown, and Baxter 1994; Schwartzman 1997; Fitzpatrick 2011).

Even so, there is much to defend in peer review. I like that in our business, authors have to have their work reviewed by readers who are, if not strangers, at least chosen by an editor with vision for a publication and not the author. I like that there is a difference between self-publication and other kinds of publication. Authors are not necessarily always the best judges of their own ideas or their own prose (and I would include myself in this group). But clearly, it is time for reform.

So how do we keep some of the good parts of the system while changing the problematic ones? One of the main features of peer review, and one of the most criticized, is the unfortunately-named “double-blind” process, where authors and reviewers do not know one another’s identities. I say it is unfortunately named because it partakes of all sorts of problematic constructs of blindness and sight. It oversells the importance of vision, and it assumes that the blind can’t identify things, which is patently ridiculous. These notions have been well criticized by Georgina Kleege (2005) in her analysis of the “hypothetical blind man” of philosophy, but “hypothetical” (as she calls him) also wreaks havoc in our scholarly world as well. We assume certain things of “blind” review that are untrue in practice. Many of us have been able to guess the identity of the author we are “blind” reviewing, and many of us have been able to guess the identity of our “blinded” reviews. Blindness is supposed to indicate objectivity, where the words on the page signify for just what they are. But of course they never do: we all have our critical hobby-horses, our preferred and, well, not preferred approaches to theory and method.

There are many proposals to transform institutions of peer review, and perhaps in time some will come to fruition. But there is a simple step that every concerned reviewer could take, right now, to make the process better, fairer, more useful and more human.

Reviewers, sign your reviews, and tell your editors that you decline anonymity.

This would have a few immediate effects. First, authors could read the reviews against the reviewer’s work, which would help them understand where the comments are coming from. Second, they would also get to see who is performing gatekeeping functions for the journal or press, which would help them evaluate it as a potential publication outlet and make editors accountable for their decisions. Third, it would make reviewers responsible not only for their words but for their tone since their names would be attached to it. This would result in fewer irresponsible reviews dashed off that are of little help to the author, and fewer “seek and destroy” operations. It would pressure reviewers to more carefully consider the author’s standpoint. It might even, in some cases, nurture some of that solidarity that Richard Rorty says scientists have over humanists as a result of their consensus on first principles.

I have begun the process over the past year and largely been successful. It is easy with book manuscripts where you know the authors’ names—you can just send them the review. With journals it’s a little trickier. And I have found that journal editors have wildly differing interpretations of what the “ethics” of disclosure might be, under what conditions I am allowed to attach my name to the review, and so forth. That in itself is revealing because it shows there is no clear consensus on the advantages or usefulness of the anonymization process. My choice has had no effect on my acceptance/rejection rate, but it has led to some interesting conversations. It has also forced me to be both fair and careful in my reviews. I think I was before, but now even moreso. Perhaps there are also people out there silently cursing me, I don’t know. But the beauty of tenure is that it doesn’t matter as much.

Of course the mere proposition of authors signing their reviews is a tremendous pain in the ass for everyone involved. Editors aren’t going to want to reveal the names of their reviewers because of the likely flak from authors. Reviewers aren’t going to want to spend the extra time or be accountable for saying “no” on a journal submission or book project (despite the fact that we do that all the time to students who know very well who we are). Authors may underestimate the value of a review from a person they don’t know. And reviewers without the security of tenure may not want their names out there in case vindictive senior colleagues get wind of their rejections.

My proposal is not a panacea, and it certainly won’t solve lots of other problems in publishing. It probably introduces lots of problems I haven’t even considered yet (which is why this is a blog post and why I won’t be responsible for anything bad that happens as a result of my proposal). I’m not even 100% convinced it’s a good idea. But naming has a way of imposing responsibility, and that seems like a good reason to give it a try. We could just limit it to tenured faculty to begin with and see how it works.

And unlike so many fixes to the peer-review system, we can try it out right now. All you have to do is sign your reviews.

Blair, Carole, Julie Brown, and Leslie Baxter. 1994. “Disciplining the Feminine.” Quarterly Journal of Speech 80 (November): 383-409. doi:10.1080/00335639409384084.
Fitzpatrick, Kathleen. 2011. Planned Obsolescence: Publishing, Technology, and the Future of the Academy. NYU Press.
Kleege, Georgina. 2005. “Blindness and Visual Culture: An Eyewitness Account.” Journal of Visual Culture 4 (2): 179 -190. doi:10.1177/1470412905054672.
Schwartzman, Roy. 1997. “Peer Review as the Enforcement of Orthodoxy.” Southern Communication Journal 63: 69-75.

Jonathan Sterne just edited a big collection of essays entitled “The Politics of Academic Labor in Communication Studies,” where 21 authors ask us to confront and deal with the big issues we now face in a changing landscape: from defunding of universities to feed the war machine, to the politics of careers, to tyranny of powerpoint, along with a host of proposals and programs for organizing. It appears any day now in the International Journal of Communication at http://ijoc.org.

Share

Tags: , , ,

7 Responses to “ A Step Toward Fixing Peer Reviews: Sign Them ”

  1. Derek Kompare on September 23, 2011 at 1:11 PM

    Hear hear! Here’s another possible benefit: timeliness. It’s harder to keep putting off writing that review (which we’ve all done) if your name is on it. It wouldn’t turn journals into models of efficiency overnight, but it would certainly help speed things up a bit.

  2. Jason Mittell on September 23, 2011 at 4:27 PM

    Great piece, and I agree with the sentiment. I’ve signed a good number of reviews in recent years, and agree that it can lead to great benefits for future conversation & connections. But one situation where I didn’t sign a manuscript review points to an issue with this strategy, and I’m curious what Jonathan or others think.

    The manuscript was flawed in a number of ways, but one of the main problems was a weak literature review about one of the main facets of the topic (television genre theory). It so happens that I published a book about that very topic, and some of the ideas the manuscript presented as original approaches were pretty close to my own as published in that now 7-year-old book (which was not in their bibliography). In my anonymous review, I pointed this out as one of the crucial gaps in the manuscript, mentioning my book as well as a few other key works that could help develop their argument. But had I signed the review, I fear the author could have dismissed my criticism as sour grapes for being left out of a bibliography (or worse, simple defensive turf-protecting) rather than a substantial critique – after all, it’s easier to deflect criticism onto a perceived personal failing of the critic than owning up to your own flaws. And while the author might have been able to deduce it was me, at least my anonymity should have at least forced them to consider the critiques.

    So, thoughts? Is the nobility of shifting the peer review process away from anonymity worth the dangers of authors assuming the worst about the reviews (& reviewers)?

  3. Jonathan on September 24, 2011 at 10:26 AM

    Hi Jason,

    You meant they don’t already assume the reviewers are hawking their own books? I don’t know enough to say about about genre theory in TV studies, but at this point when I review stuff on sound, media history, or new media studies (or whatever other field they think I have expertise in for reviewing), I can almost always point them to other people besides me who make points I want to make. If they ARE reinventing the wheel, then it’s fair game, so long as you make clear what arguments have already been made.

  4. Nele Noppe on September 25, 2011 at 3:59 AM

    My peer review experience isn’t very extensive yet (please consider this a disclaimer), but this is a very interesting suggestion. I certainly remember desperately wanting to reveal myself to a friend who was rambling about the comments I’d written about her paper as an anonymous reviewer.

    The more I think about it, though, the more I believe that what I want both as an author and as a reviewer is not full name disclosure so much as a way to communicate about the content of a review after it’s been written. When a reviewer seems to misunderstand a point I made, what I want the most is not to know who that reviewer is, but an opportunity to ask where things went wrong – was my writing unclear, was it something else? When the abovementioned friend was going on about my anonymous comments, all I wanted was a chance to talk about this interesting paper and help in ways that would go beyond shouting my opinion at her from a safe distance and not giving her an opportunity for discussion.

    Reviewers shouldn’t be obliged to answer endless requests for clarification, of course, but a chance for an author to ask one or two questions could be incredibly helpful. And maybe reviewers also want to poke more at texts they find really intriguing. I can’t quite decide if that kind of communication would work best if it were done with real names attached at every stage, though.

    Scholastica’s proposed system for rewarding quality reviewers sounds interesting, too. http://scholasticahq.com/

  5. A step toward improving peer reviews: sign them on September 25, 2011 at 10:03 AM

    […] Read the post over at Antenna, where Jonathan Gray has been running a work/school/life series. […]

  6. dhawhee on September 25, 2011 at 6:53 PM

    great post. When I was a new assistant professor,I was asked by an editor to sign my review (as part of editorial policy), and I refused. The author of the article was a graduate student at (I was pretty sure and subsequently found out I was right) an institution with faculty who would then be evaluating me for tenure. The possibility for those vulnerabilities makes me equivocate on making this a rule.

  7. Jonathan on September 26, 2011 at 1:58 PM

    Thanks for the additional replies, Nele and Debra! (or Deb?) You are both right that the revelation isn’t the most important thing, and I also doubt that this system works well for more junior reviewers. I wouldn’t want to make it a rule without a bunch of other changes as well.

    The Scholastica model is fascinating (parts of it remind me of sites for musicians), but I do wonder what their economic model is, if they’re planning to make it a for-profit venture down the road or have an IPO — their blog says “it’s a labor of love” but I bet Starbuck’s site says that somewhere too.

    The social-networky side of it also seems skewed to the perspective of junior scholars. While it’s great if you want to build reputation, given my existing reviewing and editing load (not to mention students, postdocs, friends, etc), there are very few situations in which I would approach a journal editor and ask for more articles to review and I doubt I would want to have a reviewer profile on such a site.