Peer review practices: a content analysis of external reviews in science funding

Abstract

Abstract:The primary purpose of this study is to open up the black box of peer review and to increase its transparency, understanding, and credibility. To this end, two arguments will be presented: First, epistemic and social aspects of peer review procedures are inseparable and mutually constitutive. Second, a content analysis of written reviews indicates that certain elements of peer culture from the 17 th century are still active in the scientific community. These arguments are illustrated by a case study on the peer review practices of a national funding institution, the Swiss National Science Foundation. Based on the case study and the two arguments it will be concluded more generally that peer review procedures show a distinctive specificity to the reviewed objects (e.g. papers or proposals), the organisational format (e.g. panels or external reviewers), or the surrounding context (e.g. disciplinary or interdisciplinary). Scientists, administrators, and the public may conclude that appraising peer review procedures should not be done by way of general principals but should be based on concrete factual knowledge on the specific process under discussion.3

Criteria for assessing grant applications: a systematic review

“…In the sixth and final screening round, the full texts of the 43 studies included after the third screening round plus the 37 studies resulting from the citationbased search were closely examined and final inclusion decisions were made on the basis of all four inclusion criteria. In total, 12 studies were included in the qualitative synthesis (i.e., Abdoul et al, 2012, Guetzkow et al, 2004, Hartmann and Neidhardt, 1990, Lahtinen et al, 2005, Lamont, 2009, Pier et al, 2018, Pollitt et al, 1996, Reinhart, 2010, Schmitt et al, 2015, Thomas and Lawrence, 1991, van Arensbergen et al, 2014b, Whaley et al, 2006). Hartmann's and Neidhardt's (1990) joint article was included, but not their books (Hartmann, 1990, Neidhardt, 1988, as they contain the same criteria as the joint article.…”

Funding Research Data Infrastructures: Funding Criteria in Grant Peer Review

“…Thus, the results of our analysis may lead to a deeper understanding of the peer review process (Reinhart 2010, 317). Although there is already some research on the practices and funding criteria used for classic grant peer reviews (Hartmann and Neidhardt 1990, Langfeldt 2001, Reinhart 2009, 2010, we know quite little about the adaption of reviewing practices within funding programs like the one that is at the centre of this study. Funding criteria for the promotion of information infrastructures of research data need to include additional and different criteria compared to classical grant peer review such as…”

How peer-review constrains cognition: on the frontline in the knowledge sector

“…Indeed, I left the UK partly as a result of pressures created by the Research Assessment Exercise (RAE) and partly because I choose to work with students as, not customers, but co-constructors of understanding. As peer culture disappears ( Reinhart, 2010 ), gentlemanly self-regulation is often replaced by performance driven roles. Indeed, the globalization of knowledge is itself enough to ensure that science is no longer dominated by men of a certain class who meet in the smoking rooms of London clubs –or their 20th century counterparts (see, Snow, 2012 ).…”

“… Bornmann (2011) asks if decisions are reliably obtained, free of bias and if the results have predictive validity. He echoes, for example, Reinhart’s (2010) contrast between ‘quality assurance’ and ‘self-regulation’. On the input–output view, process-variations (e.g., pre vs. post publication review) name independent variables that ground hypotheses and models.…”