A longitudinal analysis of university rankings

Abstract

<jats:p> Pressured by globalization and demand for public organisations to be accountable, efficient, and transparent, university rankings have become an important tool for assessing the quality of higher education institutions. It is therefore important to assess exactly what these rankings measure. Here, the three major global university rankings—the Academic Ranking of World Universities, the Times Higher Education ranking and the Quacquarelli Symonds World University Rankings, are studied. After a description of the ranking methodologies, it is shown that university rankings are stable over time but that there is variation between the three rankings. Furthermore, using principal component analysis and exploratory factor analysis, we demonstrate that the variables used to construct the rankings primarily measure two underlying factors: a university reputation and its research performance. By correlating these factors and plotting regional aggregates of universities on the two factors, differences between the rankings are made visible. Last, we elaborate how the results from these analysis can be viewed in light of often voiced critiques of the ranking process. This indicates that the variables used by the rankings might not capture the concepts they claim to measure. The study provides evidence of the ambiguous nature of university rankings quantification of university performance. </jats:p>

The rank boost by inconsistency in university rankings: Evidence from 14 rankings of Chinese universities

“…The finding does not change if we vary the length of the top-k list (see results for other k values in Figure S1 of Supplementary Information). Overall, different Chinese university rankings are not very consistent, in line with the claims of other similar studies (Çakır et al, 2015;Liu et al, 2011;Selten et al, 2019;.…”

“…A top ranked element is also given a higher weight characterized by an inversely proportional function. Overall, it is a very nice measure compared with many others, and is also frequently used in some recent studies, especially in some comprehensive comparisons (Aguillo et al, 2010;Çakır et al, 2015;Selten et al, 2019). One limitation of Bar-Ilan's M measure is that its physical interpretation is not very clear.…”

“…There have been intensive studies, either qualitative (Aguillo, Bar-Ilan et al, 2010;Angelis, Bassiliades, & Manolopoulos, 2019;Chen & Liao, 2012;Moed, 2017) or quantitative (Anowar, Helal et al, 2015;Çakır et al, 2015;Robinson-García, Torres-Salinas et al, 2014;Selten, Neylon et al, 2019;Vernon et al, 2018), on the comparisons of university rankings. Nevertheless, we wish to emphasize that the techniques required for a comprehensive understanding of this question are nontrivial.…”

Approach to the Calculation of Consolidated Ratings for the Tasks of Managing Higher Education Institutions

“…Analytical processing of ratings is most often carried out in the direction of analyzing their components (factors). As a rule, they have different scales and units of measurement [14,15]. The second direction is associated with the comparison of various ratings, their averaging (compilation of secondary, consolidated ratings), including taking into account changes in their structure in dynamics [1,12,16].…”

Web of Science and Scopus are not global databases of knowledge

“…Cameron Neylon and others recently criticised these rankings, demonstrating that the data sources underpinning them are heavily biased and incomplete (5,6). This seems strange, given that we have imposed a critical dependence on them within our global scholarly ecosystem.…”