Efficient sequential and parallel algorithms for record linkage

Mamun, Abdullah Al; Tian, Mi; Aseltine, Robert H.; Rajasekaran, Sanguthevar

doi:10.1136/amiajnl-2013-002034

Cited by 17 publications

(20 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We have developed algorithms that can integrate any number of datasets with a high accuracy [2], [3]. [3] describes record linkage algorithms based on single linkage clustering and Levenshtein distance method. Recently, we have extended our work to incorporate complete linkage.…”

Section: Record Linkage Accumulates Records Of Individualsmentioning

confidence: 99%

Poster: Efficient record linkage techniques

Mamun

Aseltine

Rajasekaran

2014

2014 IEEE 4th International Conference on Computational Advances in Bio and Medical Sciences (ICCABS)

Self Cite

View full text Add to dashboard Cite

Record linkage or deduplication integrates records across multiple data sources. We propose sequential and parallel techniques for record linkage using complete linkage clustering.The key idea of these approaches is radix sorting and blocking on data attributes and producing a graph-based solution. These methods have been tested on real datasets as well as synthetic datasets. They identify records belong to individuals with almost 100% accuracy.Recent advancement of health information technology requires integration of patient medical records from a large number of health services, and educational databases. Data integration or record linkage has a lot of applications, such as analysis and evaluation of diseases, mining social networking data, finding patterns in text or image datasets, and processing genomic and protein sequences.Record linkage connects multiple datasets by identifying connections among records of those datasets. Different datasets may contain records of the same individuals. Therefore the problem of record linkage is to identify clusters of records such that each cluster contains records pertaining to a single individual only. Datasets contain a huge number of duplicate records. Sometimes errors are introduced at the time of creating, collecting, and storing records. Therefore any record linkage algorithm has to deal with identifying duplicate records and handling errors. Our proposed techniques handle introduced errors and produce clusters of records of individuals very efficiently.Record linkage accumulates records of individuals. Records belonging to the same person can be thought of as a cluster. Given a set of points from an arbitrary space, the problem of clustering is to partition them such that each part has similar points. If we think of records as points from some space, the problem of records linkage can be thought of as a clustering problem. There are different versions of clustering such as hierarchical clustering, partitioning clustering, k-means clustering, k-medians clustering, densitybased clustering, model based clustering, fuzzy clustering, etc.

show abstract

Section: Record Linkage Accumulates Records Of Individualsmentioning

confidence: 99%

Poster: Efficient record linkage techniques

Mamun

Aseltine

Rajasekaran

2014

2014 IEEE 4th International Conference on Computational Advances in Bio and Medical Sciences (ICCABS)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [18], parallel data linkage algorithms and performance results obtained with data sets scaled up to 6 million records are discussed. Further, in [19], a Web-based version of these algorithms is compared against Febrl and FRIL.…”

Section: A Data Linkage Toolsmentioning

confidence: 99%

On the Accuracy and Scalability of Probabilistic Data Linkage Over the Brazilian 114 Million Cohort

Pita

Pinto

Sena

et al. 2018

IEEE J. Biomed. Health Inform.

View full text Add to dashboard Cite

Data linkage refers to the process of identifying and linking records that refer to the same entity across multiple heterogeneous data sources. This method has been widely utilized across scientific domains, including public health where records from clinical, administrative, and other surveillance databases are aggregated and used for research, decision making, and assessment of public policies. When a common set of unique identifiers does not exist across sources, probabilistic linkage approaches are used to link records using a combination of attributes. These methods require a careful choice of comparison attributes as well as similarity metrics and cutoff values to decide if a given pair of records matches or not and for assessing the accuracy of the results. In large, complex datasets, linking and assessing accuracy can be challenging due to the volume and complexity of the data, the absence of a gold standard, and the challenges associated with manually reviewing a very large number of record matches. In this paper, we present AtyImo, a hybrid probabilistic linkage tool optimized for high accuracy and scalability in massive data sets. We describe the implementation details around anonymization, blocking, deterministic and probabilistic linkage, and accuracy assessment. We present results from linking a large population-based cohort of 114 million individuals in Brazil to public health and administrative databases for research. In controlled and real scenarios, we observed high accuracy of results: 93%-97% true matches. In terms of scalability, we present AtyImo's ability to link the entire cohort in less than nine days using Spark and scaling up to 20 million records in less than 12s over heterogeneous (CPU+GPU) architectures.

show abstract

“…Design and the evaluation of sequential and parallel record linkage algorithms are discussed in [12]. One approach implements a pipeline to concatenate, sort and block records, generating a graph of matching pairs.…”

Section: Related Workmentioning

confidence: 99%

Exploring hybrid parallel systems for probabilistic record linkage

et al. 2018

View full text Add to dashboard Cite

Record linkage is a technique widely used to gather data stored in disparate data sources that presumably pertain to the same real world entity. This integration can be done deterministically or probabilistically, depending on the existence of common key attributes among all data sources involved. The probabilistic approach is very time consuming due to the amount of records that must be compared, specifically in big data scenarios. In this paper, we propose and evaluate a methodology that simultaneously exploits multicore and multi-GPU architectures in order to perform the probabilistic linkage of large-scale Brazilian governmental databases. We present some algorithmic optimizations to ensure scalability and high accuracy. We also discuss performance results obtained with different data samples, showing that a hybrid approach outperforms other configurations, providing an average speedup of 7.9 when linking up to 20.000 million records.

show abstract

Efficient sequential and parallel algorithms for record linkage

Cited by 17 publications

References 44 publications

Poster: Efficient record linkage techniques

Poster: Efficient record linkage techniques

On the Accuracy and Scalability of Probabilistic Data Linkage Over the Brazilian 114 Million Cohort

Exploring hybrid parallel systems for probabilistic record linkage

Contact Info

Product

Resources

About