Which Type of Citation Analysis Generates the Most Accurate Taxonomy of Scientific and Technical Knowledge?

Abstract

In 1965, Price foresaw the day when a citation‐based taxonomy of science and technology would be delineated and correspondingly used for science policy. A taxonomy needs to be comprehensive and accurate if it is to be useful for policy making, especially now that policy makers are utilizing citation‐based indicators to evaluate people, institutions and laboratories. Determining the accuracy of a taxonomy, however, remains a challenge. Previous work on the accuracy of partition solutions is sparse, and the results of those studies, although useful, have not been definitive. In this study we compare the accuracies of topic‐level taxonomies based on the clustering of documents using direct citation, bibliographic coupling, and co‐citation. Using a set of new gold standards—articles with at least 100 references—we find that direct citation is better at concentrating references than either bibliographic coupling or co‐citation. Using the assumption that higher concentrations of references denote more accurate clusters, direct citation thus provides a more accurate representation of the taxonomy of scientific and technical knowledge than either bibliographic coupling or co‐citation. We also find that discipline‐level taxonomies based on journal schema are highly inaccurate compared to topic‐level taxonomies, and recommend against their use.

Same data—different results? Towards a comparative approach to the identification of thematic structures in science

“…Considerations of validity (or ''accuracy'', Boyack and Klavans 2010) led to numerous partial comparisons of approaches to field delineation and topic identification, most of which either focused on similarity measures for papers (see e.g. the review in Boyack and Klavans 2010;Klavans and Boyack 2015) or on clustering methods (Šubelj et al 2016). These papers also highlight the central problem of these exercises: Since there is no 'ground truth' available, comparisons have to use arbitrary yardsticks.…”

“…In most cases, several similarity measures are compared against one that is set as universal yardstick. Klavans and Boyack (2015) use documents with more than 100 references, while Šubelj et al (2016) analyse statistical properties of solutions and use their own expertise in scientometrics. None of these approaches indicates convergence on a shared, theoretically justified standard for establishing validity.…”

Clustering of scientific publications at the KTH Library

“…Links between publications can be calculated in different ways, as direct citations, bibliographic coupling (the number of common references between two publications) or co-citations (when two publications have been cited by the same publication). Direct citation have been used here for two reasons; 1) efficiency and, 2) they perform well compared to co-citations and bibliographic coupling for clustering of research publications (Klavans & Boyack, 2017).…”

Bibliometric Analysis of Social Robotics Research: Identifying Research Trends and Knowledgebase

“…This is known as an intercitation or direct citation. Compared to other linkage methodologies like co-citation [17] and bibliographic coupling [18], direct citation networks have been found to bring out the most accurate representation of knowledge taxonomies [19] and be better at identifying To study the position of social robotics in relation to the greater field of robotics, we also retrieved articles with the query "robot*", obtaining 200,139 articles out of which 142,587 (71.2%) were connected by references. Data were retrieved on 1 July 2017.…”

“…This is known as an intercitation or direct citation. Compared to other linkage methodologies like co-citation [17] and bibliographic coupling [18], direct citation networks have been found to bring out the most accurate representation of knowledge taxonomies [19] and be better at identifying research fronts [20]. We extracted the largest connected component, which has the most tightly connected structures of knowledge related to our research target.…”