What We Still Don't Know About Peer Review

Sabaj, Omar; Pina-Stranger, Alvaro

doi:10.3138/jsp.47.2.180

Cited by 27 publications

(19 citation statements)

References 131 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…By 'unsayable' we mean that criteria are tacit and therefore difficult or impossible to articulate and analyze (Gulbrandsen, 2000. Lastly, research on peer review is mainly interested in reliability, fairness, and predictive validity (Reinhart, 2012, Sabaj Meruane et al, 2016 and not in the criteria peers use or deem appropriate. In other words, research on peer review has ignored content validity, that is, the question of which criteria are appropriate for assessing grant applications (for a comprehensive definition of content validity, see Haynes et al, 1995).…”

mentioning

confidence: 99%

Criteria for assessing grant applications: a systematic review

Hug

Aeschbach

2020

Palgrave Commun

View full text Add to dashboard Cite

Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as 'criterion' into an evaluated entity (i.e., the object of evaluation) and an evaluation criterion (i.e., the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities, as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification), as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a 'psychological' (motivation, traits) and a 'sociological' (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.

show abstract

mentioning

confidence: 99%

Criteria for assessing grant applications: a systematic review

Hug

Aeschbach

2020

Palgrave Commun

View full text Add to dashboard Cite

show abstract

“…It is, then, well beyond the scope of this book to provide a comprehensive meta-review of the Reading Peer Review 7 secondary literature on peer review. The interested reader, though, could consult one of the many other studies that have conducted such an exercise (Bornmann 2011b;Meruane, González Vergara, and Pina-Stranger 2016;Silbiger and Stubler 2019;Weller 2001). Much, although by no means all, of this research has been critical of peer-review processes, finding our faith in the practice to be misplaced (Squazzoni 2010, 19;Sugimoto and Cronin 2013, 851-2; for more positive opinions on the process, see Bornmann 2011a;Goodman 1994;Pierie, Walvoort, and Overbeke 1996).…”

Section: The Study Of Peer Reviewmentioning

confidence: 99%

Reading Peer Review

Eve

Neylon

O’Donnell

et al. 2020

View full text Add to dashboard Cite

This Element describes for the first time the database of peer review reports at PLOS ONE, the largest scientific journal in the world, to which the authors had unique access. Specifically, this Element presents the background contexts and histories of peer review, the data-handling sensitivities of this type of research, the typical properties of reports in the journal to which the authors had access, a taxonomy of the reports, and their sentiment arcs. This unique work thereby yields a compelling and unprecedented set of insights into the evolving state of peer review in the twenty-first century, at a crucial political moment for the transformation of science. It also, though, presents a study in radicalism and the ways in which PLOS's vision for science can be said to have effected change in the ultra-conservative contemporary university. This title is also available as Open Access on Cambridge Core.

show abstract

“…Ware () concludes, from several original studies, that reviewers’ opinions often differ from each other, that is, inter‐rater reliability is weak. On the other hand, Sabaj Meruane, González Vergara, and Pina‐Stranger () remind us that high agreement among reviewers might result in redundancy or, at least, in a lack of diversity, which is not good for the science in the long run.…”

Section: Pr As Critical Appraisal From the Developmental Viewpointmentioning

confidence: 99%

Peer review in the development of academic articles: Experiences of Finnish authors in the educational sciences

Atjonen

2018

Learned Publishing

View full text Add to dashboard Cite

This research examined author experiences of the developmental feedback regarding peer review of journal articles. It focused on quite experienced authors in the educational sector of all eight universities in Finland. Data were gathered using an electronic questionnaire, which was completed by 121 respondents. The majority of authors perceived reviewer feedback as developmental and appreciated critical but constructive review. Over 60% of respondents found reviewers usually keen to improve the article and to provide pertinent critique. Less than 8% reported frequent experience of reviewers who were not experts or provided unjustified accusations. A third reported reviewer feedback that was frequently biased or based on different paradigms. The most published authors were the most satisfied with the developmental feedback, but postdoctoral researchers had received more developmental complementary feedback than any other groups. The results indicate that the peer review system remains useful for ensuring article quality, but some attention is still required to ensure developmental feedback that will help the authors enhance their manuscripts.

show abstract

What We Still Don't Know About Peer Review

Cited by 27 publications

References 131 publications

Criteria for assessing grant applications: a systematic review

Criteria for assessing grant applications: a systematic review

Reading Peer Review

Peer review in the development of academic articles: Experiences of Finnish authors in the educational sciences

Contact Info

Product

Resources

About