Cleaning Noisy and Heterogeneous Metadata for Record Linking across Scholarly Big Datasets

Abstract

Automatically extracted metadata from scholarly documents in PDF formats is usually noisy and heterogeneous, often containing incomplete fields and erroneous values. One common way of cleaning metadata is to use a bibliographic reference dataset. The challenge is to match records between corpora with high precision. The existing solution which is based on information retrieval and string similarity on titles works well only if the titles are cleaned. We introduce a system designed to match scholarly document entities with noisy metadata against a reference dataset. The blocking function uses the classic BM25 algorithm to find the matching candidates from the reference data that has been indexed by ElasticSearch. The core components use supervised methods which combine features extracted from all available metadata fields. The system also leverages available citation information to match entities. The combination of metadata and citation achieves high accuracy that significantly outperforms the baseline method on the same test dataset. We apply this system to match the database of CiteSeerX against Web of Science, PubMed, and DBLP. This method will be deployed in the CiteSeerX system to clean metadata and link records to other scholarly big datasets.

Proposed threshold-based and rule-based approaches to detecting duplicates in bibliographic database

“…The study used two datasets: CiteSeerX and IEEE. Similar research has also been conducted by Sefid et al [14] using four datasets: CiteSeerX, WoS, DBLP, and PubMed. A machine learning approach was adopted to perform entity matching using as many as seven features.…”

“…Other factors that cause duplication are typography errors [12], omitted fields, and missing values [13]. The absence of a digital object identifier (DOI) in a scientific article also causes duplication [14]. According to research by Gyawali et al [15], more than 82% of the papers collected in databases do not have a DOI.…”