Robustness of personal rankings: the Handelsblatt example

Abstract

In the last years, Handelsblatt has published several rankings of business economists from German, Swiss and Austrian research institutions based on their journal publication output. These rankings have a strong influence on the academic profession. We scrutinize the Handelsblatt methodology by examining the effect the rankings’ underlying algorithms and assumptions have on the scores and ranks of individual researchers. In doing so, we clarify how robust the result is with respect to these internal parameters. Since the parameters used by Handelsblatt are not scientifically substantiated but defined ad hoc, this question is of great importance. For each parameter variation, we provide several robustness measures for both the Handelsblatt life’s work ranking and the Handelsblatt recent research performance ranking. E.g., if one applies a weighting scheme that lays more emphasis on first tier journal publications such that the weight of a particular category is always double of the weight of the next lower category, rank correlations based on all researchers in both personal rankings exceed 80 %. However, if one solely considers the top 25 performing researchers rank correlations fall below 50 and 20 % of researchers even drop out of this top group. Further research as well as the discussion in the academic community should clarify whether these correlations verify the robustness of the ranking or manifest the opposite.

The Index Number Problem with DEA: Insights from European University Efficiency Data

“…Such avenues of inquiry are connected to the quality debate regarding evaluations and rankings for universities, as, for example, stated by Bornmann [105], Osterloh and Frey [106], and Harmann [107] for an international perspective. For the national German context regarding the VHB JOURQUAL3 journal rankings in economics and business administration, Eisend [108] and Schrader and Hennig-Thurau [109] discuss this as well as Lorenz and Löffler [110] for the Handelsblatt ranking of business economists. Policy implications point towards the discourse regarding conflicts of interest for university management in pursuing efficiency goals compared to other objectives such as excellence, reputation, or cooperation, as Blackmore points out [111].…”

Catching up or falling behind? Promising changes and persistent patterns across cohorts of economics PhDs in German-speaking countries from 1991 to 2008

“…7 Handelsblatt ranking was introduced in 2005/2006 and has since been established as a tool for evaluating individual academic economists and economics departments in the German-speaking area. Despite receiving some criticism for the incentives it creates (Hofmeister and Ursprung 2007), its subjectivity (Frey and Rost 2010), and its robustness (Lorenz and Löffler 2015), the ranking is widely applied in the German-speaking area, e.g. in recruitment decisions.…”

Citation metrics as an additional indicator for evaluating research performance? An analysis of their correlations and validity

“…e.g. Clermont and Dirksen 2016;Lorenz and Löffler 2015;Müller 2010). Due to their partial (CHE) or exclusive (Handelsblatt) use of absolute indicators, which do not consider the number of staff at the respective BuSs, the larger BuSs have a certain advantage.…”

“…The success of such an objective can only be measured indirectly, which is why different metrics and indicators are proposed and discussed in the literature. To systematize such indicators, Lorenz and Löffler (2015) distinguish between productivity, impact, and esteem indicators. Under the heading of productivity indicators, they subsume those indicators which map the publication output of an academic organizational unit (e.g.…”

“…However, such weighting of written research work is a matter of controversy (for a related overview, see Clermont and Dirksen 2016 and the literature cited therein). Since, for example, the journal rating JOURQUAL2 is used to weight the quality of journal articles, Eisend (2011) investigated the validity of this rating on the basis of a correlation study with other rankings and ratings already established in the science community, whereas Lorenz and Löffler (2015) analysed the robustness of the methodology of the Handelsblatt rankings.…”