The impact of imbalanced training data on machine learning for author name disambiguation

Abstract

In supervised machine learning for author name disambiguation, negative training data are often dominantly larger than positive training data. This paper examines how the ratios of negative to positive training data can affect the performance of machine learning algorithms to disambiguate author names in bibliographic records. On multiple labeled datasets, three classifiers -Logistic Regression, Naïve Bayes, and Random Forestare trained through representative features such as coauthor names, and title words extracted from the same training data but with various positive-negative training data ratios. Results show that increasing negative training data can improve disambiguation performance but with a few percent of performance gains and sometimes degrade it. Logistic Regression and Naïve Bayes learn optimal disambiguation models even with a base ratio (1:1) of positive and negative training data. Also, the performance improvement by Random Forest tends to quickly saturate roughly after 1:10 ~ 1:15. These findings imply that contrary to the common practice using all training data, name disambiguation algorithms can be trained using part of negative training data without degrading much disambiguation performance while increasing computational efficiency. This study calls for more attention from author name disambiguation scholars to methods for machine learning from imbalanced data.

Effect of forename string on author name disambiguation

“…The impact of forename string on author name disambiguation is measured on four labeled data sets that have been widely used in many studies, separately or jointly (for example, Cota et al, ; Ferreira et al, ; Kim & Kim, ; Momeni & Mayr, ; Müller et al, ; Pereira et al, ; Santana, Gonçalves, Laender, & Ferreira, ; Shin et al, ; Wu, Li, Pei, & He, ; Zhu et al, ).…”

“…PENN: Labeled for Han et al () by researchers at the Pennsylvania State University, this data set was originally comprised of 8,453 name instances with their coauthorship, article title, and venue information. As its original version contained duplication and labeling errors, several studies modified the data set before use (for example, Cota et al, ; Kim & Kim, ; Santana et al, ; Shin et al, ). This study reuses one of recent revisions by Kim () in which 5,018 name instances and their associated metadata are linked to DBLP records after deduplication and verification of correctness…”

“…Two name instances in a block are compared for similarity over these four features as follows. Each name string is lower‐cased, converted into ASCII format, and segmented into an array of 2–4‐gram, following several studies (Han, Xu, Zha, & Giles, ; Kim & Kim, ; Kim, Kim, & Owen‐Smith, ; Louppe et al, ; Treeratpituk & Giles, ). For example, “Mark” is converted into a list of “ma,” “ar,” “rk,” “mar,” “ark,” and “mark.” Then a cosine similarity of the term frequency (TF) between the 2–4‐gram lists of two name instances is calculated as a forename similarity score for the instance pair.…”

“…This prediction score is used as a similarity distance between the pair to be fed into a hierarchical agglomerative clustering algorithm, which groups name instances into a cluster if their distances are above a certain threshold. Following previous studies (Kim & Kim, ; Levin et al, ; Liu et al, ; Louppe et al, ; Torvik & Smalheiser, ), this threshold is decided by trying various distance values between 0 and 1 in each block and choosing one that produces the best clustering result (measured by B‐Cubed F1 explained below) for the block . Meanwhile, to evaluate how a heuristic performs in comparison with algorithmic disambiguation, name instances in test data that match on all available name (surname + forename) strings are assumed to refer to the same author.…”

“…These clusters predicted by a disambiguation method from a test data set are compared to truth clusters that are generated from labels of name instances in the same test data set. Disambiguation performance is evaluated by B‐Cubed (B 3 ), following previous studies (Delgado, Martínez, Montalvo, & Fresno, ; Kim & Kim, ; Levin et al, ; Louppe et al, ; Momeni & Mayr, ; Müller et al, ; Qian, Zheng, Sakai, Ye, & Liu, ). This measure consists of three parts—B 3 Recall ( R ), B 3 Precision ( P ), and B 3 F ( F )—defined as follows (Levin et al, ): Here, t is a name instance in truth clusters T .…”