Efficient Record Linkage Algorithms Using Complete Linkage Clustering

Abstract

Data from different agencies share data of the same individuals. Linking these datasets to identify all the records belonging to the same individuals is a crucial and challenging problem, especially given the large volumes of data. A large number of available algorithms for record linkage are prone to either time inefficiency or low-accuracy in finding matches and non-matches among the records. In this paper we propose efficient as well as reliable sequential and parallel algorithms for the record linkage problem employing hierarchical clustering methods. We employ complete linkage hierarchical clustering algorithms to address this problem. In addition to hierarchical clustering, we also use two other techniques: elimination of duplicate records and blocking. Our algorithms use sorting as a sub-routine to identify identical copies of records. We have tested our algorithms on datasets with millions of synthetic records. Experimental results show that our algorithms achieve nearly 100% accuracy. Parallel implementations achieve almost linear speedups. Time complexities of these algorithms do not exceed those of previous best-known algorithms. Our proposed algorithms outperform previous best-known algorithms in terms of accuracy consuming reasonable run times.

The emerging landscape of health research based on biobanks linked to electronic health records: Existing resources, statistical challenges, and potential opportunities

“…Additionally, we must consider issues such as data processing and rules for linking records for a single patient. Many statistical methods have been developed for linking records corresponding to individual patients across data sources, and many of these methods explicitly address issues of privacy . Statistical methods have also been developed for combining data across distributed data sources where data from individual patients are not accessible .…”

“…Many statistical methods have been developed for linking records corresponding to individual patients across data sources, and many of these methods explicitly address issues of privacy. [223][224][225][226][227] Statistical methods have also been developed for combining data across distributed data sources where data from individual patients are not accessible. 228,229 Yang et al developed methods for performing meta-analysis based on existing GWAS, and similar methods should be developed for PheWAS studies in the future.…”

The Emerging Landscape of Epidemiological Research Based on Biobanks Linked to Electronic Health Records: Existing Resources, Analytic Challenges and Potential Opportunities

“…Many statistical methods have been developed for linking records corresponding to individual subjects across data sources, and many of these methods explicitly address issues of privacy. [237][238][239][240][241] Statistical methods have also been developed for combining data across distributed data sources where data from individual subjects is not accessible, called distributed regression analysis. These methods involve sharing sufficient statistics of the data (functions of the individual-level data) from which the individual-level data are not recoverable.…”

Exploring hybrid parallel systems for probabilistic record linkage

“…The parallel version was tested with a 6 million records synthetic data set, presenting a linear speedup varying from 7.5 to 26.4 (from 8 to 32 cores, respectively). These methods are deeper explained in [11] and a Web-based version, called RLT-S, is discussed and compared to other existing free tools (Febrl and FRIL) in [10].…”