The Content of My Peer Reviews Did Not Change Once I Started Signing Them

I am a strong proponent of signed reviews, the practice of reviewers disclosing their identities by including their names in their peer reviews. I have explained my reasoning in a previous blog post and editorial when discussing open science practices at Emerging Adulthood. It is my personal opinion that any potential costs associated with the practice are outweighed by the benefits afforded through openness and transparency. People seem quick to cite the potential for retaliation, especially for early career scholars and those from under-represented communities. I am honestly confused by this response because it a) assumes the worst of our colleagues and b) as Hilda Bastian nicely articulated, if indeed there are instance of retaliation then as a community we should address the perpetrators directly rather than cower in fear. Moreover, masked review does not prevent retaliation, as authors may be certain they know who the reviewers are and may treat them accordingly, even though they may be wrong. Signing reviews brings these dynamics all into the open.

All of that said, my feelings and beliefs are likely to be of little interest to most people. Rather, what we all want are some data that speak to the issue. There have been some studies on the topic, but they are all limited in some way. For example, van Sambeek and Lakens (2020) found that authors were more likely to sign their reviews when making positive recommendations (accept/minor revisions) compared with negative recommendations. The majority of the data, however, came from manuscripts that were ultimately published, and thus their analyses were based on a biased sample. Moreover, they only examined the recommendation made by the editor or the reviewers, and not the content of the reviews. In a self-report survey study, Lynam et al. (2019) found that respondents believed that signing their reviews would lead to reviews that are less critical, more civil, and take longer to complete, all with rather small effects.

These studies collectively rely on self-report, a biased sample of reviews, or incomplete information about the content of the reviews. To address these limitations, I dug into my own archives to analyze the content of my reviews. Since my first peer review in 2008 I have kept a word file of every review I have completed. I am sure that I am missing some, but any missing reviews would not be due to any kind of systematic bias, rather just simple lack of organization. Whereas of course there are many limitations to analyzing reviews from a single person, there are also many benefits. Indeed, this analysis is a direct within-person behavioral test of the patterns reported by Lynam et al.

My archive includes 203 reviews completed between 2008 and 2019. I started signing my reviews in 2015, which resulted in 147 unsigned review and 56 signed reviews. I have fewer signed reviews because around the time I started signing is also when I became an Associate Editor and then Editor, and thus conducted a smaller number of ad-hoc reviews for other journals. My editorial letters are not included in this analysis, as those are a different beast altogether.

I subjected my reviews to the Linguistic Inquiry and Word Count (LIWC) program to analyze the content of my reviews, and whether the content changed as a function of whether or not I signed the reviews. My version of LIWC is on the older side, so it is based on the 2007 dictionary. The data file includes everything from the dictionary, although clearly some features are more relevant than others, so I do not describe all of the data here. I only examine four features of the reviews that I thought were most relevant: word count, positive emotion words, negative emotion words, and cognitive mechanism words. The full LIWC results, R code, and the LIWC 2007 dictionary are freely available at osf.io/uf63k/. The raw reviews are not included in that data file since some people might feel that is a violation of privacy and the peer review process, but I have an identical file that includes the text of the reviews, which I am happy to send upon request.

I was interested in two questions:

Did signing my reviews lead to a change in the content of the reviews?

The answer is a pretty clear no. There were no differences between unsigned and signed reviews in terms of the length of my reviews (t = 0.41, p = .68), positive emotion words (t = -0.96, p = .33), negative emotion words (t = 1.07, p = .29), or cognitive mechanism words (t = -0.49, p = .62). These non-differences are clear in the violin plots:

It is worth noting, I think, that the frequency of positive emotion words is higher than the frequency of negative emotion words.

Signing my reviews is confounded with time; I started signing at a specific time and have signed almost every review since, with the exception of a couple I did not sign when I co-reviewed with a student. Thus, in many ways not-signed/signed is a binary version of what is a potentially interesting continuous variable: time.

Did the content of my reviews change over time?

The answer here is also a pretty clear no. For the most part. There was a negative correlation between time and word count (r = -.22, p = .002), indicating my reviews have gotten a little shorter over time. This is consistent with oft-quoted remark that length of review is negatively correlated with time in the field. However, looking at the scatterplot below shows that the association is quite noisy. The correlations between time and positive emotion words (r = .05, p = .45), negative emotion words (r = .002, p = .98), and cognitive mechanism words (r = .05, p = .48) were all very small.

So there you have it. Signing my reviews did not seem to change the content much, at least with respect to these few indicators that I examined. Yes, I could have analyzed more data, or analyzed these data more appropriately, but I did this work while avoiding several other more pressing tasks so was not looking to put in maximal effort. The data are available, dear reader, so have at it!