Methods to evaluate institutional responses to performance‐based research funding systems

Abstract

Performance‐based research funding systems (PBRFS) have been introduced in many countries for allocating funding to research institutions. There continues to be considerable debate about the effectiveness and consequences of these systems. This paper suggests several methods that can be applied to evaluate how institutions respond to new incentives created by the introduction of a PBRFS. The methods are illustrated using longitudinal data from the New Zealand PBRFS, which assesses institutional performance and allocates funds based on individual researcher performance, although not all methods require individual data. Longitudinal data enable the identification of entry, exit and quality transformation of researchers and the contribution of these dynamics to changes in university and discipline research quality. The approach enables a deeper understanding of individual and institutional responses to PBRFSs, the sustainability of changes and the contributions of changes in researcher quality and discipline composition to changes in institutional performance.

What is Behind the Shine? The Dark Side of Research Evaluation: A Conceptual Framework

“…The great interest of Russian authors in RE in recent years (Figure 4) stems from the intro of a country-specific research evaluation system for prioritizing science [33], [38], [48], [54]. The increased interest of authors from the UK, New Zealand and the USA (Figure 4) can be explained by the constant improvements in their RE systems introduced there long ago [17], [44], [47].…”

“…Second, the ineffective research evaluation system can be identified as the main reason for the emergence of the RE dark effects. Typically, ineffectiveness arises when research evaluation systems are not subject-area/country-specific ( [4], [16], [19], [20], [26], [32], [33], [35], [41], [42]) or are not balanced to specific public needs ( [12], [15], [19], [26], [31], [38], [42]), do not account for the available scientific infrastructure ( [3], [8], [32], [35], [46]) or place too much emphasis on quantity rather than quality ( [14], [15], [17], [19], [20], [23], [30], [31], [32], [34], [36], [37], [48]).…”

“…Poorly formulated, implemented or distorted science policy, university or national, is another important cause of the RE dark effects ( [10], [33], [38], [55]). Other reasons that can be pointed out address university, publisher or funding management issues ( [7], [8], [14], [17], [22], [31], [33], [44]), insufficient scientific infrastructure ( [3], [11], [35], [46]), poorly structured or outdated research systems ( [8], [15], [22], [26], [30], [32], [34], [35], [38]), subject behaviour issues ( [4], [5], [23], [27], [30], [32], [37], [39], [46]), etc.…”

A scoping review on what constitutes a good research culture

“…(1,2) There is a strong connection between concerns about research culture and the inappropriate use of metrics and indicators that drive both institutional and individual researcher behavior, assessment and reward. (3)(4)(5) In response to these concerns, a number of actions have emerged to enable and encourage the adoption of a healthier research culture. (6)(7)(8) International action to address the underlying drivers of poor research culture include INORMS SCOPE framework for responsible research evaluation (9) ; Declaration on Research Assessment (DORA) (10) ; development of 10 principles for the measurement of research performance: the Leiden Manifesto for Research Metrics (11) ; establishment of the International School on Research Impact Assessment (ISRIA) (12) ; and the HuMetricsHSS Initiative.…”

“…As highlighted by the Wellcome Trust and others, there remains a tendency for quantity over quality, assessment based on publications, journal prestige and funding. (5,15,23,24) Any attempts on reform requires commitment from everyone (e.g., publishers, research institutions, funders, researchers etc.) so that diversity, impact, teamwork, open research, and assessment systems are valued.…”