Mixed Messages: Referees' Comments on the Manuscripts They Review

Abstract

There are many criticisms of the manuscript review process but few systematic studies of the referees' comments on manuscripts they review. The authors examined comments on a sample of first submission manuscripts reviewed by the American Sociological Review (ASR), 1977–1981. Positive and negative comments were classified into twelve categories: topic, theory, review of literature, design, data, sample, measurement, analysis, results, style, ad hominem, general. No manuscripts received unequivocally favorable reviews, but some reviews were less negative than others. Referees were more critical of manuscripts in comments to the editor than in comments to authors. Authors did not see comments to the editor nor did they know the referees' recommendations concerning their manuscripts. Thus, editors had more information than authors, and authors may complain that they got mixed messages from two or more referees. However, there were surprisingly few actual contradictions in referees' comments. While most comments (both positive and negative) were general, a substantial number were about theory and analysis. Reviews of the same manuscript by different referees yielded surprisingly few contradictions, most of those about theory. The biggest difference between the comments by referees recommending publication and by those recommending rejection was praise. Referees were equally critical of all manuscripts, but were more likely to praise those recommended for publication.

Impact of bibliometrics upon the science system: Inadvertent consequences?

“…In the review of articles for journals it is the other way around: Reviewers agree more about acceptance than about rejection of articles (CICHETTI, 1991;BAKANIC et al, 1989). Others have pointed to the considerable role of extraneous factors such as sheer luck or being well integrated into the right networks, or belonging to the right institution (COLE et al, 1981).…”

Reviewers' Views on Reviewing: An Examination of the Peer Review Process in Criminal Justice

“…) Other social scientists, however, have argued that correspondence between reviewers' recommendations is low (Bornstein, 1991;Cicchetti, 1991;Lindsey, 1991Lindsey, , 1988. In a review of reviews of manuscripts for the American Sociological Review for 1977-1981Bakanic, McPhail and Simon (1989 reported that no papers received unequivocally favorable reviews. Cicchetti (1991) reported that studies of reviewer agreement for manuscripts at 11 social science journals ranged from 0.19 to 0.54 (eight of the 11 journals had agreement rates of less than 0.4) and for six medical journals reviewer agreement rates ranged from 0.26 to 0.37.…”

“…Bakanic, McPhail and Simon (1989), using reviews submitted on 755 manuscripts considered by American Sociological Review, reported that only 11% of all manuscripts received any contradictory reviews. (Also see Eberley, 1987;Hargens, 1988;Hargens & Herting, 1990a, b.…”

“…Comments written for return to authors and comments written for editors only do vary, and as Bakanic et al (1989, p. 639) reported, "Referees were more critical of manuscripts in comments to the editor than in comments to authors". Bakanic et al (1989) reviewed reviewers' comments on 393 accepted manuscripts and 361 randomly selected rejected manuscripts submitted to American Sociological Review between 1977 and 1981. They found that negative comments "far outnumbered" positive comments, regardless of whether a manuscript was published or not.…”

Polaridad de los comentarios y consistencia interna en los informes de arbitraje de artículos de investigación

“…De forma opuesta, si un informe presenta una alta tasa de comentarios positivos y la recomendación es rechazar el artículo, estaremos entonces frente a un informe con baja consistencia. La consistencia es muy importante desde el punto de vista de los autores del artículo, ya que, como han demostrado Bakanic y otros (2), estos suelen recibir mensajes contradictorios, que pueden resultar confusos, especialmente para los investigadores que se encuentran al inicio de su vida académica. Esta situación contradice lo planteado por Jefferson y otros (3), quienes señalan que lo deseable es que el proceso permita a los autores mejorar sus manuscritos.…”