A letter on Ancaiani et al. ‘Evaluating scientific research in Italy: the 2004-10 research evaluation exercise’

Abstract

This letter documents some problems in Ancaiani et al. (2015). Namely the evaluation of concordance, based on Cohen's kappa, reported by Ancaiani et al. was not computed on the whole random sample of 9,199 articles, but on a subset of 7,597 articles. The kappas relative to the whole random sample were in the range 0.07-0.15, indicating an unacceptable agreement between peer review and bibliometrics. The subset was obtained by non-random exclusion of all articles for which bibliometrics produced an uncertain classification; these raw data were not disclosed, so that concordance analysis is not reproducible. The VQR-weighted kappa for Area 13 reported by Ancaiani et al. is higher than that reported by Area 13 panel and confirmed by Bertocchi et al. (2015), a difference explained by the use, under the same name, of two different set of weights. Two values of kappa reported by Ancaiani et al. differ from the corresponding ones published in the official report. Results reported by Ancaiani et al. do not support a good concordance between peer review and bibliometrics. As a consequence, the use of both techniques introduced systematic distortions in the final results of the Italian research assessment exercise. The conclusion that it is possible to use both technique as interchangeable in a research assessment exercise appears to be unsound, by being based on a misinterpretation of the statistical significance of kappa values.

On the agreement between bibliometrics and peer review: Evidence from the Italian research assessment exercises

“…Two of the authors of the present paper documented the flaws of EXP1 and contested the interpretation of data as indicative of a substantial agreement [6][7][8][9]. The present paper takes advantage of the recent availability of the raw data of the two experiments, in order to deepen the analysis and reach conclusive results on issues that had remained open due to the sole availability of aggregated data.…”

“…The unique exception-confirmed by a statistical meta-analysis of the data-was Economics and Statistics, for which the protocol of the experiment was substantially modified with respect to the other fields. Baccini and De Nicolao [8,9] also raised some questions on the sampling protocol used for EXP1, which are considered in details also in this paper.…”

“…In order to reproduce ANVUR's results, the sets of linear weights and VQR-weights are solely considered. In addition, for improving readability, the analysis and the comments are limited to the computation based on VQR-weights, while the results for linear weights are available as S1 Table. At first, the estimation of (3), (5) and 7are considered for the agreement of the bibliometric and peer-review ratings by means of the estimators (4), (6) and (8). The estimation was carried out for each area and for the global population in both EXP1 and EXP2.…”

Are Italian research assessment exercises size-biased?

“…In order to validate the mixed approach of peer review and bibliometric assessment introduced by VQR 2004-2010, a representative sample of scientific articles in this exercise was experimentally submitted for both evaluations. Ancaiani et al 2015report evidence of a significant degree of concordance between the two methods, a conclusion which is however challenged in Baccini and De Nicolao (2017), and defended in Sergio et al (2017c).…”

Untitled

“…Tables 1 and 2 reports the sizes of the sub-samples for EXP1 and EXP2, respectively. The exclusion of the IR papers might have boosted the value of the agreement measures, as argued by Baccini and De Nicolao [8]. The conjecture sounds as follows.…”