Peer Review vs Metric-based Assessment: Testing for Bias in the RAE Ratings of UK Economics Departments

Abstract

RAE ratings have been criticized as biased in favour of universities that are old, in England, large and represented on the panel. We investigate these accusations for the 1996 and 2001 RAE ratings of economics departments using independent rankings from the academic literature as quality controls. We find RAE ratings to be largely in agreement with the profession's view of research quality as documented by independent rankings, although the latter appear to be more focused on research quality at the top end of academic achievement. Accusations of bias find no support in the data, except for panel membership in 1996.

Formulating research policy on expert advice

“…Recent studies of Research Assessments (e.g., Taylor 2011, Butler and McAllister, 2009, and Clerides et al, 2011 find that when a University department is represented by one of its academics on a panel of experts its standing improves. According to these authors, such findings are suggestive of bias.…”

“…Second, objective measures of research quality do not fully explain expert assessments. See for example, Coupé (2001), Butler and McAllister (2009), Clerides et al (2011), and Taylor (2011) for the UK, Reale et al (2007) for Italy, and Aksnes and Taxt (2004) for Norway. According to these authors, such findings are suggestive of expert bias and related shortcomings in expert assessments.…”

“…Common across all countries with national assessments is the delegation of the task to measure research quality to expert committees consisting of local academics. Typically, such groups of experts are granted an extraordinary degree of discretion, the proceedings of their deliberations are not transparent, and their recommendations are not subject to oversight (Williams, 1998;Clerides et al, 2011).…”

Are leading papers of better quality? Evidence from a natural experiment

“…Other scientists support the subjective assessments by panels of experts, despite the fact that self-interest of experts might bias their assessment (Clerides et al, 2008) or that experts might use suboptimal sources of information when forming their judgement. Oswald (2007) for example suggests experts should be using the number of cites of articles rather than journal titles as quality indicators.…”

A Tale of Two Metrics: Research Assessment vs Recognised Excellence

“…These papers review and produce a meta-ranking of journals that takes into account the audience for which the ranking is targeted (Hudson, 2013), evaluate the role of citations in conducting quality evaluations (Laband, 2013), and discuss how peer review panels should combine output counts with citations to generate an overall view of quality (Sgroi and Oswald, 2013). Other literature has looked at research assessment more generally, investigating accusations of bias (Clerides et al, 2011), and a lack of stability in rankings when the balance of quality/quantity weightings and citations systems change (Frey and Rost, 2010) 3 . We use the Hudson meta-ranking in our analysis, but we do not address citations for reasons we outline below.…”