Benchmarking scientific output in the social sciences and humanities: The limits of existing databases

Abstract

The goal of this paper is to examine the impact of linguistic coverage of databases used by bibliometricians on the capacity to effectively benchmark the work of researchers in social sciences and humanities. We examine the strong link between bibliometrics and the Thomson Scientific's database and review the differences in the production and diffusion of knowledge in the social sciences and humanities (SSH) and the natural sciences and engineering (NSE). This leads to a re-examination of the debate on the coverage of these databases, more specifically in the SSH. The methods section explains how we have compared the coverage of Thomson Scientific databases in the NSE and SSH to the Ulrich extensive database of journals. Our results show that there is a 20 to 25% overrepresentation of English-language journals in Thomson Scientific's databases compared to the list of journals presented in Ulrich. This paper concludes that because of this bias, Thomson Scientific databases cannot be used in isolation to benchmark the output of countries in the SSH.

The competitive advantage of nations: An application to academia

“…However, since this is true for all countries, it should not necessarily bias comparison between countries. In the Social Sciences, low ISI coverage is caused by the fact that ISI over-represents English language journals (Archambault et al 2006), whereas much of the output in the Social Sciences has traditionally appeared in books and national journals in the local language (Nederhof, 2006). Therefore, although the exclusive use of ISI data might not be appropriate to analyse research productivity for individual academics in the Social Sciences, we decided that coverage was sufficient enough to include the three Social Science categories (Social Sciences General, Economics & Business, and Psychology & Psychiatry) in our broad, country-level, analyses.…”

Can alternative indicators overcome language biases in citation counts? A comparison of Spanish and UK research

“…Nevertheless, these databases (especially WoS) are biased towards literature in English by including a greater proportion of academic literature in English than in other languages. The major citation databases are also biased towards journal articles (Albarillo 2014;Archambault et al 2006, Orduña-Malea et al 2014Van Leeuwen et al 2001). This is a major drawback for humanities research and, to a lesser extent, social science research, because these have stronger national and regional orientations, tending to be published in languages other than English and often in books (Engels et al 2012;Nederhof 2006).…”

The future of research assessment in the humanities: bottom-up assessment procedures

“…An SSH approach towards bibliometrics and scientometrics Bibliometric analyses face many problems when applied to SSH disciplines (Nederhof et al, 1989;Archambault et al, 2006;Nederhof, 2006;van Leeuwen, 2013). However, Hammarfelt (2016: 115) observes a shift from investigating coverage issues towards studying the characteristics of SSH publication practices and developing bibliometric approaches sensitive to the organization of SSH research fields.…”