Research evaluation and journal quality weights

Abstract

Research evaluations based on quality weighted publication output are often criticized on account of the employed journal quality weights. This study shows that evaluations of entire research organizations are very robust with respect to the choice of readily available weighting schemes. We document this robustness by applying rather different weighting schemes to otherwise identical rankings. Our unit of analysis consists of German, Austrian and Swiss university departments in business administration and economics.

Robustness of personal rankings: the Handelsblatt example

“…In this respect, our result is conform to Krapf (2011) who has already claimed that the actual rank assigned by Handelsblatt is not suitable (and not intended to be suitable) for evaluating individual researchers. However, ''managing by numbers'' is still common, and Germany's academic community is not the only one to have raised concerns about the danger of a mechanistic use of ratings when comparing applicants' profiles [for the UK see Hussain (2011)].…”

“…This approach assumes equivalence between all rank neighbors which implies that, e.g., the difference between the first and second rank is equivalent to the difference between ranks 100 and 101. The above-mentioned study Krapf (2011) uses Spearman's rho. However, due to the features we just mentioned, we are not convinced that it is appropriate for our purposes.…”

“…The study most similar to ours is that of Krapf (2011). Krapf uses alternative journal rankings (e.g., Jourqual 2, impact factors, journal-rating of the Wirtschaftsuniversität Wien) and compares the resulting department and life's work rankings to the corresponding Handelsblatt rankings by providing correlation measures.…”

You get what you ‘pay’ for: academic attention, career incentives and changes in publication portfolios of business and economics researchers

“…It contains information on researchers' journal article output as measured in co-author-and qualityadjusted Handelsblatt points (see Krapf 2011 for the details) 2 , as well as information on year of PhD, age, gender, and discipline (business administration vs. economics). These data were collected via the online research monitoring portal initialized by the German Economic Association ("Portal Forschungsmonitoring") and quality-approved by the Thurgau Institute of Economics and the Konjunkturforschungsstelle KOF at ETH Zurich respectively.…”

The Index Number Problem with DEA: Insights from European University Efficiency Data

“…To fulfil this task effectively, deciders have to rely on information regarding the performance of universities, recently, for example, provided by a growing number of national and international university rankings [1][2][3][4]. Rankings have evolved regarding their principal setup, incorporating criticism addressing indicators, institutional inclusion, and data quality, including the discourse on journal publication and the individual researcher level [5][6][7][8]. This also went hand in hand with an increased influence on policies and resource decisions in higher education [9][10][11][12].…”