Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise

Abstract

During the Italian research assessment exercise, the national agency ANVUR performed an experiment to assess agreement between grades attributed to journal articles by informed peer review (IR) and by bibliometrics. A sample of articles was evaluated by using both methods and agreement was analyzed by weighted Cohen’s kappas. ANVUR presented results as indicating an overall “good” or “more than adequate” agreement. This paper re-examines the experiment results according to the available statistical guidelines for interpreting kappa values, by showing that the degree of agreement (always in the range 0.09–0.42) has to be interpreted, for all research fields, as unacceptable, poor or, in a few cases, as, at most, fair. The only notable exception, confirmed also by a statistical meta-analysis, was a moderate agreement for economics and statistics (Area 13) and its sub-fields. We show that the experiment protocol adopted in Area 13 was substantially modified with respect to all the other research fields, to the point that results for economics and statistics have to be considered as fatally flawed. The evidence of a poor agreement supports the conclusion that IR and bibliometrics do not produce similar results, and that the adoption of both methods in the Italian research assessment possibly introduced systematic and unknown biases in its final results. The conclusion reached by ANVUR must be reversed: the available evidence does not justify at all the joint use of IR and bibliometrics within the same research assessment exercise

Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?

“…The conclusion by Wilsdon et al (2015) is also reached by Baccini and De Nicolao (2016), who challenge the data and findings of ANVUR (2013) on the agreement between peer review and bibliometrics in the evaluation of products within the second Italian research assessment exercise, VQR 2004-2010. The authors explain that "the degree of agreement has to be interpreted, for all research fields, as unacceptable, poor or, in a few cases, as, at most, fair.…”

The north–south divide in the Italian higher education system

“…To the best of our knowledge, there are no other studies on the subject with the only exception of an istitutional report (which is not a study) by the Italian Agency for the Evaluation of University and Research (ANVUR, 2014). The report aggregates at territorial level the results of the national research assessment exercise VQR 2004-2010, which has been strongly criticized (Franco, 2013;Baccini and De Nicolao, 2016;Baccini, 2016;Abramo, D'Angelo, and Di Costa, 2014a; in terms of methodology and performance indicators (one for all, the overall assessement of universities is based on three products only per professor over a seven year period).…”

A letter on Ancaiani et al. ‘Evaluating scientific research in Italy: the 2004-10 research evaluation exercise’

“…Anyhow, the agreement between peer review and bibliometrics, as proxied by linear-weighted and VQRweighted kappas, is still poor to fair in Areas 1-9, while the result in Area 13 is questionable due to changes introduced in the protocol with respect to the one adopted in all the other areas (Baccini andDe Nicolao 2016a, 2016b). In fact, Ancaiani et al write: "The value of K ranges from 0.16 to 0.61 depending on the area and weights, being on average equal to 0.32, a value that is usually considered as 'poor to fair'".…”