Filling in the gaps: The interpretation of <i>curricula vitae</i> in peer review

Kaltenbrunner, Wolfgang; Rijcke, Sarah de

doi:10.1177/0306312719864164

Cited by 31 publications

(17 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…According to our synthesis, which is mostly based on studies on project funding, peers evaluate the applicant in terms of a resource needed to implement the proposed project (instrumental role). In fellowship programs, however, peers evaluate the applicant also in order to decide whether she/he is suited for a further step in her/his academic career (Kaltenbrunner and de Rijcke, 2019). Lastly, our findings depend on the choices made in the included studies, such as the number of reported criteria, the level of abstraction of the reported criteria, or the words used to describe the criteria.…”

Section: Discussionmentioning

confidence: 91%

“…Rather, we understand it as a repertoire from which peers choose when PALGRAVE COMMUNICATIONS | https://doi.org/10.1057/s41599-020-0412-9 REVIEW ARTICLE PALGRAVE COMMUNICATIONS | (2020) 6:37 | https://doi.org/10.1057/s41599-020-0412-9 | www.nature.com/palcomms assessing grant applications. In addition, we conceive the evaluation criteria (and evaluated entities) as situated concepts (Yeh and Barsalou, 2006) that are (re)shaped by the actual assessment in which they are enacted (Kaltenbrunner and de Rijcke, 2019).…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Criteria for assessing grant applications: a systematic review

Hug

Aeschbach

2020

Palgrave Commun

View full text Add to dashboard Cite

Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as 'criterion' into an evaluated entity (i.e., the object of evaluation) and an evaluation criterion (i.e., the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities, as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification), as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a 'psychological' (motivation, traits) and a 'sociological' (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.

show abstract

Section: Discussionmentioning

confidence: 91%

Section: Discussionmentioning

confidence: 99%

Criteria for assessing grant applications: a systematic review

Hug

Aeschbach

2020

Palgrave Commun

View full text Add to dashboard Cite

show abstract

“…According to our synthesis, which is mostly based on studies on project funding, peers evaluate the applicant in terms of a resource needed to implement the proposed project (instrumental role). In fellowship programs, however, peers evaluate the applicant also in order to decide whether she/he is suited for a further step in her/his academic career (Kaltenbrunner & de Rijcke 2019). Lastly, our findings depend on the choices the included studies have made, such as the number of reported criteria, the level of abstraction of the reported criteria, or the words used to describe the criteria.…”

Section: Discussionmentioning

confidence: 90%

Criteria for assessing grant applications: A systematic review

Hug¹,

Aeschbach²

2019

Preprint

View full text Add to dashboard Cite

Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use in assessing grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce the Scriven Model, which separates each criterion into an evaluated entity (i.e. the object of the evaluation) and an evaluation criterion (i.e. the dimension along which an entity is evaluated). In total, this synthesis includes 12 studies. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities as well as the relations between them. Based on a network analysis, we propose a conceptualization that groups the identified evaluation criteria and evaluated entities into aims, means, and outcomes. We compare our results to criteria found in studies on research quality and guidelines of funding agencies. Since peer review is often approached from a normative perspective, we discuss our findings in relation to two normative positions, the fairness doctrine and the ideal of impartiality. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.

show abstract

“…While this provides him with a robust typology of appraisal devices, it does not let him pay sufficient attention to what I call practices of appraisal devices – that is, the processes by which actors engage with and interpret appraisal devices, including how they decide the criteria and means for using or disputing them in concrete situations. Indeed, the devices theory has recently been criticized for its reductive and functionalistic tendencies when applied in studies of academic evaluation (Kaltenbrunner and de Rijcke 2019 ). However, focusing on practices of appraisal devices, this paper tackles such criticism by emphasizing the reflexive work required by actors using them.…”

Section: Theorizing Practices Of Appraisal Devicesmentioning

confidence: 99%

Navigating Uncertainty: Early Career Academics and Practices of Appraisal Devices

Nästesjö

2020

Minerva

View full text Add to dashboard Cite

There is a lack of objective evaluative standards for academic work. While this has been recognized in studies of how gatekeepers pass judgment on the works of others, little is known about how scholars deal with the uncertainty about how their work will be evaluated by gatekeepers. Building upon 35 interviews with early career academics in political science and history, this paper explores how junior scholars use appraisal devices to navigate this kind of uncertainty. Appraisal devices offer trusted and knowledgeable appraisals through which scholars are informed whether their work and they themselves are good enough to succeed in academia. Investigating how early career academics rely upon appraisals from assessors (i.e., ‘academic mentors’), the study adds to existing literature on uncertainty and worth in academic life by drawing attention to how scholars’ anticipatory practices are informed by trusting the judgment of others. The empirical analysis demonstrates that early career academics are confronted with multiple and conflicting appraisals that they must interpret and differentiate between. However, the institutional conditions for dealing with uncertainty about what counts in future evaluations, as well as which individuals generally come to function as assessors, differ between political science and history. This has an impact on both valuation practices and socialization structures. Focusing on what I call practices of appraisal devices, the paper provides a conceptual understanding of how scholars cope with uncertainties about their future. Furthermore, it expands existing theory by demonstrating how scholars’ self-concept and desired identities are key to the reflexive ways appraisal devices are used in the course of action.

show abstract

Filling in the gaps: The interpretation of curricula vitae in peer review

Cited by 31 publications

References 43 publications

Criteria for assessing grant applications: a systematic review

Criteria for assessing grant applications: a systematic review

Criteria for assessing grant applications: A systematic review

Navigating Uncertainty: Early Career Academics and Practices of Appraisal Devices

Contact Info

Product

Resources

About