Disagreement and Agonistic Chance in Peer Review

Abstract

The purpose of grant peer review is to identify the most excellent and promising research projects. However, sociologists of science and STS scholars have shown that peer review tends to promote solid low-risk projects at the expense of more original and innovative projects that often come with higher risk. It has also been shown that the review process is affected by significant measures of chance. Against this background, the aim of this study is to theorize the notions of academic judgment and agonistic chance and to present and analyze situations in which expert reviewers are faced with the challenge of trying to decide which grant proposals to select when there is strong disagreement. The empirical analysis is based on ethnographic observations of ten panel groups at the Swedish Research Council in the areas of natural and engineering sciences. By focusing on disagreement, the study provides a more in-depth understanding of how agonistic chance creeps into the peer-review process and becomes part of the consensus that is created.

Random grant allocation from the researchers’ perspective: Introducing the distinction into legitimate and illegitimate problems in Bourdieu’s field theory

“…It was presumably no coincidence that other interviewees also repeatedly used the word ‘crazy’ (01, physics, professor, 107; 12, biology, professor, 52; 19, physics, postdoc, 83; 20, physics, postdoc, 68; 21, biology, postdoc, 90; 27, physics, predoc, 42) to denote those research ideas that would have a better chance of being funded through a grant lottery (see also Roumbanis, 2021). ‘Crazy’ meant that the ideas clearly departed from previous research traditions and programs.…”

“…(03, physics, professor, 58) I think [...] the random selection process can in fact break up structures, and that's what science thrives on -that we don't just always think in fixed patterns, that we instead want to break up structures, that we want to do new things. (16, biology, postdoc, 150) It was presumably no coincidence that other interviewees also repeatedly used the word 'crazy ' (01, physics, professor, 107; 12, biology, professor, 52; 19, physics, postdoc, 83; 20, physics, postdoc, 68; 21, biology, postdoc, 90; 27, physics, predoc, 42) to denote those research ideas that would have a better chance of being funded through a grant lottery (see also Roumbanis, 2021). 'Crazy' meant that the ideas clearly departed from previous research traditions and programs.…”

Luck and the ‘situations’ of research

“…His argument that studying luck can ‘balance skewed neoliberal narratives about individual efforts and achievement’ (p. 209) resonates with other recent accounts of the ‘myth’ of meritocracy and the tendency of the successful to underestimate privilege in their success ( Frank, 2016 ; Power et al, 2016 ). Other work theorizing luck has sought to disaggregate luck, chance, and serendipity by defining the different forms these can take ( Yaqub, 2018 ): Both Roumbanis (2021) and Sand and Jongsma (2020) , for instance, discuss different types of chance or luck, in terms that are tied to the contexts in which they emerge.…”

Towards Theorizing Peer Review

“…Severin and Chataway (2021a) explored causes and effects of reviewer overburdening and found that scholars, referees, editors, and publishers believe that the reviewing workload has become unbearable, that the workload is distributed unequally among scholars, and that overburdening is caused by an increase in manuscript submissions, insufficient editorial triage, lack of reviewing instructions, difficulties in recruiting reviewers, inefficient manuscript handling, and a lack of institutionalization of peer review. Roumbanis (2021) theorized how panels struggle to reach a consensus decision when there is strong disagreement among panelists. He calls the results of such struggles agonistic chance, that is, unforeseen consequences of social interactions in peer review.…”