Record linkage approaches in big data: A state of art study

El-Ghafar, Randa M. Abd; Gheith, Mervat; El-Bastawissy, Ali; Nasr, Eman S.

doi:10.1109/icenco.2017.8289792

Cited by 10 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using the CA in scenarios(2,3,4,5,7) achieved better results than a single WA (scenario 1) in terms of efficiency and effectiveness. As fig.4illustrates, the usage of CA (scenarios 2, 3, 4, 5, 7) in the different scenarios achieved better performance time compared to the WA (first scenario).…”

mentioning

confidence: 95%

“…Blocking techniques depend on distributing the big dataset into several small blocks such that elements that reside in the same block are more likely to be matched. Many blocking techniques have been used in literature as clarified in [4].…”

Section: Introductionmentioning

confidence: 99%

“…They can utilize commodity hardware and provide fault tolerance, high scalability, and flexibility to distributed processing. MR is an early popular framework because of its simplicity and easiest of programming however, it has several limitations [4]. In addition, every single job has to read the input data from Hadoop Distributed File System (HDFS), process it, and write it back to HDFS.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Effective Entity Resolution Approach for Big Data

El-Ghafar¹,

El-Bastawissy²,

Nasr³

et al. 2021

IJITEE

Self Cite

View full text Add to dashboard Cite

Entity Resolution (ER) is defined as the process 0f identifying records/ objects that correspond to real-world objects/ entities. To define a good ER approach, the schema of the data should be well-known. In addition, schema alignment of multiple datasets is not an easy task and may require either domain expert or ML algorithm to select which attributes to match. Schema agnostic meta-blocking tries to solve such a problem by considering each token as a blocking key regardless of the attributes it appears in. It may also be coupled with meta-blocking to reduce the number of false negatives. However, it requires the exact match of tokens which is very hard to occur in the actual datasets and it results in very low precision. To overcome such issues, we propose a novel and efficient ER approach for big data implemented in Apache Spark. The proposed approach is employed to avoid schema alignment as it treats the attributes as a bag of words and generates a set of n-grams which is transformed to vectors. The generated vectors are compared using a chosen similarity measure. The proposed approach is a generic one as it can accept all types of datasets. It consists of five consecutive sub-modules: 1) Dataset acquisition, 2) Dataset pre-processing, 3) Setting selection criteria, where all settings of the proposed approach are selected such as the used blocking key, the significant attributes, NLP techniques, ER threshold, and the used scenario of ER, 4) ER pipeline construction, and 5) Clustering where the similar records are grouped into the similar cluster. The ER pipeline could accept two types of attributes; the Weighted Attributes (WA) or the Compound Attributes (CA). In addition, it accepts all the settings selected in the fourth module. The pipeline consists of five phases. Phase 1) Generating the tokens composing the attributes. Phase 2) Generating n-grams of length n. Phase 3) Applying the hashing Text Frequency (TF) to convert each n-grams to a fixed-length feature vector. Phase 4) Applying Locality Sensitive Hashing (LSH), which maps similar input items to the same buckets with a higher probability than dissimilar input items. Phase 5) Classification of the objects to duplicates or not according to the calculated similarity between them. We introduced seven different scenarios as an input to the ER pipeline. To minimize the number of comparisons, we proposed the length filter which greatly contributes to improving the effectiveness of the proposed approach as it achieves the highest F-measure between the existing computational resources and scales well with the available working nodes. Three results have been revealed: 1) Using the CA in the different scenarios achieves better results than the single WA in terms of efficiency and effectiveness. 2) Scenario 3 and 4 Achieve the best performance time because using Soundex and Stemming contribute to reducing the performance time of the proposed approach. 3) Scenario 7 achieves the highest F-measure because by utilizing the length filter, we only compare records that are nearly within a pre-determined percentage of increase or decrease of string length. LSH is used to map the same inputs items to the buckets with a higher probability than dis-similar ones. It takes numHashTables as a parameter. Increasing the number of candidate pairs with the same numHashTables will reduce the accuracy of the model. Utilizing the length filter helps to minimize the number of candidates which in turn increases the accuracy of the approach.

show abstract

mentioning

confidence: 95%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Effective Entity Resolution Approach for Big Data

El-Ghafar¹,

El-Bastawissy²,

Nasr³

et al. 2021

IJITEE

Self Cite

View full text Add to dashboard Cite

show abstract

“…Some research on algorithms that address the computational burden of the comparison and classification tasks in record linkage has been undertaken. Most work on distributed and parallel algorithms for record linkage is specific to the MapReduce paradigm [ 15 ], a programming model for processing large data sets in parallel on a cluster. Few sources detail the comparison and classification tasks themselves, with the focus on load balancing algorithms to address issues associated with data skew.…”

Section: Introductionmentioning

confidence: 99%

“…The blocking techniques used in these studies are based on the same techniques used for traditional probabilistic and deterministic linkages [ 15 ]. There are many blocking techniques used in these conventional approaches to record linkages that reduce the comparison space significantly, even when running a linkage on a single machine [ 26 ].…”

Section: Introductionmentioning

confidence: 99%

Secure Record Linkage of Large Health Data Sets: Evaluation of a Hybrid Cloud Model

Brown

Randall

2020

JMIR Med Inform

View full text Add to dashboard Cite

Background The linking of administrative data across agencies provides the capability to investigate many health and social issues with the potential to deliver significant public benefit. Despite its advantages, the use of cloud computing resources for linkage purposes is scarce, with the storage of identifiable information on cloud infrastructure assessed as high risk by data custodians. Objective This study aims to present a model for record linkage that utilizes cloud computing capabilities while assuring custodians that identifiable data sets remain secure and local. Methods A new hybrid cloud model was developed, including privacy-preserving record linkage techniques and container-based batch processing. An evaluation of this model was conducted with a prototype implementation using large synthetic data sets representative of administrative health data. Results The cloud model kept identifiers on premises and uses privacy-preserved identifiers to run all linkage computations on cloud infrastructure. Our prototype used a managed container cluster in Amazon Web Services to distribute the computation using existing linkage software. Although the cost of computation was relatively low, the use of existing software resulted in an overhead of processing of 35.7% (149/417 min execution time). Conclusions The result of our experimental evaluation shows the operational feasibility of such a model and the exciting opportunities for advancing the analysis of linkage outputs.

show abstract

Towards a Record Linkage Layer to Support Big Data Integration

Kruse

2019

Business Information Systems Workshops

View full text Add to dashboard Cite

Record linkage approaches in big data: A state of art study

Cited by 10 publications

References 15 publications

An Effective Entity Resolution Approach for Big Data

An Effective Entity Resolution Approach for Big Data

Secure Record Linkage of Large Health Data Sets: Evaluation of a Hybrid Cloud Model

Towards a Record Linkage Layer to Support Big Data Integration

Contact Info

Product

Resources

About