Measuring academic influence: Not all citations are equal

Abstract

The importance of a research article is routinely measured by counting how many times it has been cited. However, treating all citations with equal weight ignores the wide variety of functions that citations perform. We want to automatically identify the subset of references in a bibliography that have a central academic influence on the citing paper. For this purpose, we examine the effectiveness of a variety of features for determining the academic influence of a citation.By asking authors to identify the key references in their own work, we created a dataset in which citations were labeled according to their academic influence. Using automatic feature selection with supervised machine learning, we found a model for predicting academic influence that achieves good performance on this dataset using only four features.The best features, among those we evaluated, were features based on the number of times a reference is mentioned in the body of a citing paper. The performance of these features inspired us to design an influence-primed h-index (the hip-index). Unlike the conventional h-index, it weights citations by how many times a reference is mentioned. According to our experiments, the hip-index is a better indicator of researcher performance than the conventional h-index.

Functions of Uni- and Multi-citations: Implications for Weighted Citation Analysis

“…Tang and Safer (2008) found that giving high importance to multi-citations may help improve citationbased rankings. Zhu et al (2015) also found that in-text citation frequency was the best feature to help spot citations that were considered crucial to the authors of a citing paper, and that frequency-weighted citation ranking can outperform traditional citation ranking of top authors, at least in the research field they studied.…”

“…Studies have experimented with weighting citations by the frequency with which they occur in the text (e.g. Ding et al, 2013;Hou, Li, & Niu, 2011;Tang & Safer, 2008;Zhu et al, 2015), by the citation impact of citing papers (Ding & Cronin, 2011), and by the location and context in which they are cited (Boyack, Small, & Klavans, 2013;Jeong, Song, & Ding, 2014). It has been found that frequency-weighted citation rankings can outperform traditional citation rankings of top authors, and that in-text citation frequency was the best of many full-text features to help spot citations that were considered crucial to the citing papers by their authors (Zhu et al, 2015).…”

Scientific Auditing Firms

“…We have, however, a powerful set of tools that have only recently come into existence, namely, a digitized scientific corpus and the techniques of modern data science [5][6][7][8][9][10][11][12][13][14]. For analyzing the scientific corpus, relevant data science techniques include citation network analysis, natural language processing, and many other statistical methods developed for the processing of large data sets.…”

Dimensions and uncertainties of author citation rankings: Lessons learned from frequency‐weighted in‐text citation counting

“…By weighing citations by their frequency of appearance in the text of a scientific paper, it is hoped that essential Both authors contributed equally to this article. Zhu, Turney, Lemire, and Vellino (2014) recently found that, at least in the field that they studied, in-text citation frequency was the best of many other full-text features to help spot citations that were considered crucial to the citing papers by their authors. This could improve traditional citation analysis significantly because a high incidence of perfunctory citations has been observed (Small, 1982).…”

“…Increasingly available digital full-text documents along with advances in technologies for text processing have now made it feasible to conduct large-scale studies on weighted citation counting. It has been found that frequency-weighted citation ranking can outperform traditional citation ranking of top authors in a hard-science field (Zhu et al, 2014). Such studies have experimented with weighing citations by the frequency with which they are referred to in the text (e.g., Ding, Liu, Guo, & Cronin, 2013;Hou, Li, & Niu, 2011;Zhu, Turney, Lemire, & Vellino, 2014) or by the citation impact of citing papers (Ding & Cronin, 2011) or by weighing cocitations by the distances between locations in the text where they are cited (Boyack, Small, & Klavans, 2013) or by text similarities of in-text citation contexts (Jeong, Song, & Ding, 2014).…”