One Step Closer Towards Responsible Research Assessment in Psychology: Evaluation and Testing of a New Tool in Practice

Abstract

Quantitative metrics like the Journal Impact Factor, citation counts and the h-index have been criticized heavily for being invalid measures of research quality. Nevertheless, they are still commonly used in research assessment. In recent years, numerous initiatives have emerged, seeking to fundamentally change the way research assessment is conducted by emphasizing a more holistic assessment based on research quality rather than quantity. My goal was to test and evaluate a tool proposed by Gärtner et al. (2022) for a more responsible research assessment in practice. In the current study 63 papers nominated by 21 voluntarily participating researchers were rated according to the new scheme. Then, inter-rater-reliability, associations between the new and traditional indicators and feedback from the participants on the new tool were examined. Discussions arising from intensively working with the scheme led to some modifications of the scheme and to the creation of a precise manual, which aims to improve objectivity in the future. Inter-rater reliability for the three raters varied between the different items of the scheme. Besides a negative association between the new indicators and the h-index, no associations were found. The feedback from participants revealed the importance of transparency concerning the scheme. This study brings the proposed tool one step closer to being implemented in practice and is therefore an important contribution to a more responsible research assessment.