Does bibliometric research confer legitimacy to research assessment practice? A sociological study of reputational control, 1972-2016

Abstract

The use of bibliometric measures in the evaluation of research has increased considerably based on expertise from the growing research field of evaluative citation analysis (ECA). However, mounting criticism of such metrics suggests that the professionalization of bibliometric expertise remains contested. This paper investigates why impact metrics, such as the journal impact factor and the h-index, proliferate even though their legitimacy as a means of professional research assessment is questioned. Our analysis is informed by two relevant sociological theories: Andrew Abbott’s theory of professions and Richard Whitley’s theory of scientific work. These complementary concepts are connected in order to demonstrate that ECA has failed so far to provide scientific authority for professional research assessment. This argument is based on an empirical investigation of the extent of reputational control in the relevant research area. Using three measures of reputational control that are computed from longitudinal inter-organizational networks in ECA (1972–2016), we show that peripheral and isolated actors contribute the same number of novel bibliometric indicators as central actors. In addition, the share of newcomers to the academic sector has remained high. These findings demonstrate that recent methodological debates in ECA have not been accompanied by the formation of an intellectual field in the sociological sense of a reputational organization. Therefore, we conclude that a growing gap exists between an academic sector with little capacity for collective action and increasing demand for routine performance assessment by research organizations and funding agencies. This gap has been filled by database providers. By selecting and distributing research metrics, these commercial providers have gained a powerful role in defining de-facto standards of research excellence without being challenged by expert authority.

Professional standards in bibliometric research evaluation? A meta-evaluation of European assessment practice 2005–2019

“…In a recent study, [19] investigated the academic research area of evaluative citation analysis as the academic sector that is closely aligned with bibliometric evaluation practice. Abstract knowledge can also be stored in specialized artefacts, which Abbott refers to as expert commodities.…”

“…Research assessment is potentially within the jurisdiction of professional experts who can define the nature of assessment problems, and offer solutions that effectively address clients' needs. One recent paper presents an empirical investigation of whether the academic research area of 'evaluative citation analysis' has successfully defined scientific standards for bibliometric research evaluation during the period 1972-2016 [19]. Based on organizational network analysis and the theory of intellectual fields as reputational organizations, [19] concluded that the field of evaluative citation analysis has been characterized by low levels of reputational control, evidenced by high shares of outsider contributions and new actors entering the field throughout the examined period.…”

“…One recent paper presents an empirical investigation of whether the academic research area of 'evaluative citation analysis' has successfully defined scientific standards for bibliometric research evaluation during the period 1972-2016 [19]. Based on organizational network analysis and the theory of intellectual fields as reputational organizations, [19] concluded that the field of evaluative citation analysis has been characterized by low levels of reputational control, evidenced by high shares of outsider contributions and new actors entering the field throughout the examined period. They argued that this lack of reputational control within the academic research area is consistent with observed difficulties in establishing scientific authority for bibliometric assessment practice.…”

“…1. Expert organizations and individual experts having a central position in the academic research area of evaluative citation analysis were identified by prior research [19]. Requests for non-confidential evaluation studies were sent in early 2015 to experts from 35 research organizations in 13 European countries.…”

“…2. Evaluation studies were extracted from a set of WoS publications identified as "follow-up research on citation impact indicators" [19]. This set comprised all publications in WoS that cite any of 169 specified citation impact indicators, including a total of 2757 publications from 2005-2014 and 2776 publications from 2015-2019.…”