Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology

Abstract

Traditional metric indicators of scientific productivity (e.g., journal impact factor; h-index) have been heavily criticized for being invalid and fueling a culture that focuses on the quantity, rather than the quality, of a person’s scientific output. There is now a wide-spread demand for specified alternatives to current academic evaluation practices. In a previous report, we laid out four basic principles of a more responsible research assessment in academic hiring and promotion processes (Schönbrodt et al., 2022). The present paper offers a specific proposal for how these principles may be implemented in practice: We argue in favor of broadening the range of relevant research contributions and thus propose concrete quality criteria (including ready-to-use online templates) for published research articles, data sets and research software. These criteria are supposed to be used primarily in the first phase of the assessment process. Their function is to help establish a minimum threshold of methodological rigor that candidates need to pass in order to be further considered for hiring and promotion. In contrast, the second phase of the assessment process will focus more on the actual content of candidates’ research output and necessarily use more narrative means of assessment. We hope that this proposal will help get our colleagues in the field engaged in a discussion over ways of replacing current invalid evaluation criteria with ones that relate more closely to scientific quality.

Responsible Research Assessment I: Implementing DORA and CoARA for hiring and promotion in psychology

“…Unfortunately, these vital features of research have played a very minor role in research assessment so far. We argue that this needs to change, and that they have to be moved to the forefront instead (Leising et al, 2022).…”

“…However, in the remainder of the present paper, we do focus on the Research dimension, because this is the area in which an urgent need for alternative evaluation criteria has been most clearly articulated for many years (Abele-Brehm & Bühner, 2016;European Commission, Directorate-General for Research and Innovation, 2022;Leising et al, 2022;LERU, 2022) 2 . We suggest that three kinds of research contributions should be considered by hiring and promotion committees: (a) journal articles, (b) published data sets, and (c) research software.…”

“…Methodological rigor, including open research practices, goes a long way in establishing these properties, and preliminary evidence suggests that assessing this merit is possible in a reliable fashion (for concrete suggestions for measurable quality indicators, see Fig. 1; Gärtner et al, 2022;Leising et al, 2022aLeising et al, , 2022b.…”

“…Furthermore, the use of relatively distal quantitative measures such as the JIF, the h-index, or simply the quantity of publications in hiring and promotion may have the unintended side-effect of fueling a "publish or perish" culture in which the use of questionable research practices is incentivized. This risk is significant, given the high incentive value of attaining a permanent position in academia (Leising et al, 2022) and the fact that, at the same time, academia is largely lacking effective mechanisms of quality control and self-correction (Vazire & Holcombe, 2022).…”

“…We propose four principles of responsible research assessment applicable to the hiring and promotion of individual researchers, and a proposal for a two-phase assessment procedure for hiring committees that combines the objectivity and efficiency of indicators with a qualitative and narrative assessment of a shortlist of candidates. In a separate document, a specific, actionable way of implementing these principles is proposed (Gärtner et al, 2022). Supplemental material can be found at https://osf.io/4wynr/.…”