Critical comments on Institute for Scientific Information impact factors: a sample of inorganic molecular chemistry journals

Abstract

In this paper, empirical data are analysed to show some of the problems involved in the use of the Institute for Scientific Information’s (ISI) impact factors (IFs). Based on earlier work of the authors, and elaborating on some new topics, the paper shows that IFs as defined by ISI have shortcomings which make them inappropriate for the purposes for which people use them: researchers for their publication strategy, policy makers (at different levels) to evaluate research performance, and librarians to evaluate their journal collections. Whereas earlier papers have focused on problems involved with the definitions of the constituting elements of the classical IF and the resulting errors, this paper focuses on the problems related to other characteristics of scientific journals; in particular, the influence of the distribution of papers among document types in a journal, the effects of splitting of journals or changing their names, the measurement of (un)citedness of papers in a journal, and the chosen length of the citation window within the definition of the classical IF. This will raise the fundamental question of whether an indicator, based on only a one- to two-year citation window, will be sufficiently valid to be of any use in analyses of journal and research performance.

The impact of Cochrane Reviews: a mixed-methods evaluation of outputs from Cochrane Review Groups supported by the National Institute for Health Research

“…It takes a number of years for citation counts to peak, typically 3 or 4 years post publication, and a window of 5 years has been suggested as most appropriate for research assessment. 90,91 Indeed, it may take up to 10 years for the full impact of research to be apparent. 92 This was a retrospective evaluation of impact and as such may be a greater risk of recall bias than one in which data is collected prospectively.…”

The use and misuse of journal metrics and other citation indicators

“…The pluses and minuses of the impact factor, as viewed by various bibliometricians, are widely discussed in the literature (Leydesdorff 2008;Moed 2002;Moed 2005a;Moed and van Leeuwen 1995;Moed et al 1999;Moed et al 2004;van Leeuwen et al 1999;van Leeuwen and Moed 2002;van Leeuwen and Moed 2005;Zitt and Small 2008).…”

On the ratio of citable versus non-citable items in economics journals

“…As a result, journals containing a high proportion of review articles tend to have higher JIFs than other journals which GARFIELD (1996) also notes. VAN LEEUWEN et al (1999) recommends each document type being treated separately to take into account that document types are not cited the same.…”