Entity deduplication in big data graphs for scholarly communication

Abstract

PurposeSeveral online services offer functionalities to access information from “big research graphs” (e.g. Google Scholar, OpenAIRE, Microsoft Academic Graph), which correlate scholarly/scientific communication entities such as publications, authors, datasets, organizations, projects, funders, etc. Depending on the target users, access can vary from search and browse content to the consumption of statistics for monitoring and provision of feedback. Such graphs are populated over time as aggregations of multiple sources and therefore suffer from major entity-duplication problems. Although deduplication of graphs is a known and actual problem, existing solutions are dedicated to specific scenarios, operate on flat collections, local topology-drive challenges and cannot therefore be re-used in other contexts.Design/methodology/approachThis work presents GDup, an integrated, scalable, general-purpose system that can be customized to address deduplication over arbitrary large information graphs. The paper presents its high-level architecture, its implementation as a service used within the OpenAIRE infrastructure system and reports numbers of real-case experiments.FindingsGDup provides the functionalities required to deliver a fully-fledged entity deduplication workflow over a generic input graph. The system offers out-of-the-box Ground Truth management, acquisition of feedback from data curators and algorithms for identifying and merging duplicates, to obtain an output disambiguated graph.Originality/valueTo our knowledge GDup is the only system in the literature that offers an integrated and general-purpose solution for the deduplication graphs, while targeting big data scalability issues. GDup is today one of the key modules of the OpenAIRE infrastructure production system, which monitors Open Science trends on behalf of the European Commission, National funders and institutions.

We Can Make a Better Use of ORCID: Five Observed Misapplications

“…In this context, ORCID is integrated as an "inverted list" of bibliographic records, each bearing ORCID iDs for the authors that claimed that record via their ORCID profile. All such records are harmonised and deduplicated (Manghi et al, 2020b) so as to build one bibliographic record out of the many describing the same research product but collected from different sources. Of particular interest to this investigation, is the fact that such "richer" metadata records, feature author/creator metadata which may bear a list of ORCID iDs, as collected from ORCID records and as collected from data sources (ORCID referrals).…”

Data Mining for Non-Redundant Big Data Using dynamic KMEAN

“…Many prominent clustering algorithms, such as k-means and k-median, use the number of clusters to output as an input. In de-duplication applications, this information is unknown [7] [8].…”

FDup: a framework for general-purpose and efficient entity deduplication of record collections

“…Clustering makes sure potentially equivalent records are grouped into clusters (aka “blocks”), within which the pair-wise similarity match will be quadratically applied. Blocking reduces the number of matches but, most importantly, allows for the parallel execution of the process across different blocks ( Manghi et al, 2020 ). The sliding window technique further optimizes the number of matches within the individual blocks by sorting the records in such a way that similar records are likely kept close to each other and then matching each record with the “k” following records (“K-length window”).…”