Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods

Abstract

Ranking of research institutions by bibliometric methods is an improper tool for research performance evaluation, even at the level of large institutions. The problem, however, is not the ranking as such. The indicators used for ranking are often not advanced enough, and this situation is part of the broader problem of the application of insufficiently developed bibliometric indicators used by persons who do not have clear competence and experience in the field of quantitative studies of science. After a brief overview of the basic elements of bibliometric analysis, I discuss the major technical and methodological problems in the application of publication and citation data in the context of evaluation. Then I contend that the core of the problem lies not necessarily at the side of the data producer. Quite often persons responsible for research performance evaluation, for instance scientists themselves in their role as head of institutions and departments, science administrators at the government level and other policy makers show an attitude that encourages 'quick and dirty' bibliometric analyses whereas better quality is available. Finally, the necessary conditions for a successful application of advanced bibliometric indicators as support tool for peer review are discussed.

Impact of SciELO and MEDLINE indexing on submissions to Jornal de Pediatria

“…Em primeiro lugar, muitos já apontaram que contar citações é uma forma falha de avaliar o mérito acadêmico, entre outros porque não se mede a citação a artigos específi-cos [20][21][22][23][24][25] . Em segundo lugar, se um artigo é citado com freqüência como exemplo de pesquisa de má qualidade, mesmo assim a menção repetida a esse artigo irá aumentar o fator de impacto da revista.…”

Towards polarisation-encoded quantum key distribution in optical fibre networks

“…Assessment and comparison of journals according to their impact factors is a well-established approach in scientometrics 14,15 , despite the limitations and shortcomings 16 . The impact factor of a journal is defined as the quotient between the number of times an article published in the journal in the previous 2 years is cited in the year of observation and the number of articles this journal has published in these previous 2 years.…”

A relational database for bibliometric analysis

“…A comprehensive review of problems concerning data quality can be found in (Hood/Wilson 2003). There are several publications which emphasize the aspect of data quality for bibliometric analyses e. g. (Fernández et al 1993;van Raan 2005). I will not expand on the topic of data quality here any further, since it is more a problem of data management than a genuine part of bibliometrics.…”