Untitled

Cota, Ricardo G.; Ferreira, Anderson A.; Nascimento, Cristiano; Gonçalves, Marcos André; Laender, Alberto H. F.

doi:10.1002/asi.v61:9

Cited by 57 publications

(1 citation statement)

References 408 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this example, the Pearson correlation between xn and yn may be as high as 0.7-0.8. In other words, the fact that xn and yn may show a high correlation may be completely explained by the common factor n. A similar observation has already been made before in bibliometrics (West and Bergstrom, 2010), and related concerns were already raised by Pearson as early as 1896.…”

Section: B Size-dependent and Size-independent Perspectivessupporting

confidence: 69%

Systematic analysis of agreement between metrics and peer review in the UK REF

Traag

Waltman

2019

Palgrave Commun

View full text Add to dashboard Cite

When performing a national research assessment, some countries rely on citation metrics whereas others, such as the UK, primarily use peer review. In the influential Metric Tide report, a low agreement between metrics and peer review in the UK Research Excellence Framework (REF) was found. However, earlier studies observed much higher agreement between metrics and peer review in the REF and argued in favour of using metrics. This shows that there is considerable ambiguity in the discussion on agreement between metrics and peer review. We provide clarity in this discussion by considering four important points: (1) the level of aggregation of the analysis; (2) the use of either a size-dependent or a size-independent perspective; (3) the suitability of different measures of agreement; and (4) the uncertainty in peer review. In the context of the REF, we argue that agreement between metrics and peer review should be assessed at the institutional level rather than at the publication level. Both a size-dependent and a size-independent perspective are relevant in the REF. The interpretation of correlations may be problematic and as an alternative we therefore use measures of agreement that are based on the absolute or relative differences between metrics and peer review. To get an idea of the uncertainty in peer review, we rely on a model to bootstrap peer review outcomes. We conclude that particularly in Physics, Clinical Medicine, and Public Health, metrics agree quite well with peer review and may offer an alternative to peer review.

show abstract

Section: B Size-dependent and Size-independent Perspectivessupporting

confidence: 69%