Efficient and Effective Duplicate Detection in Hierarchical Data

Leitão, Luís; Calado, Pável; Herschel, Melanie

doi:10.1109/tkde.2012.60

Cited by 28 publications

(29 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our experiments are performed on four different datasets 3 , two synthetic datasets [12] (Country and Company) with sampled spelling errors and two real datasets (Restaurant and Tungsten). The Country and Company datasets contain 9 and 11 fields/features respectively.…”

Section: Settingsmentioning

confidence: 99%

See 1 more Smart Citation

Extending Naive Bayes Classifier with Hierarchy Feature Level Information for Record Linkage

Zhou

Howroyd

Danicic

et al. 2015

Advanced Methodologies for Bayesian Networks

View full text Add to dashboard Cite

Abstract. Probabilistic record linkage has been well investigated in recent years. The Fellegi-Sunter probabilistic record linkage and its enhanced version are commonly used methods, which calculate match and non-match weights for each pair of corresponding fields of record-pairs. Bayesian network classifiers -naive Bayes classifier and TAN have also been successfully used here. Very recently, an extended version of TAN (called ETAN) has been developed and proved superior in classification accuracy to conventional TAN. However, no previous work has applied ETAN in record linkage and investigated the benefits of using a naturally existing hierarchy feature level information. In this work, we extend the naive Bayes classifier with such information. Finally we apply all the methods to four datasets and estimate the F1 scores.

show abstract

Section: Settingsmentioning

confidence: 99%

“…Also we will extend the naive Bayes classifier (referred to as HR-NBC) by introducing hierarchy restrictions between features. As discussed in previous work [11,12], these hierarchy restrictions are very useful to avoid unnecessary computation of field comparison, and to help refine the Bayesian network structure.…”

Section: Introductionmentioning

confidence: 99%

Extending Naive Bayes Classifier with Hierarchy Feature Level Information for Record Linkage

Zhou

Howroyd

Danicic

et al. 2015

Advanced Methodologies for Bayesian Networks

View full text Add to dashboard Cite

show abstract

“…Surveys [8,9]. review the various approaches, including named attributes computations [5], schema mapping [2,17] and duplicate detection in hierarchical data [10], all which inform the construction of profile linkage techniques.…”

Section: Record Linkage and Entity Resolutionmentioning

confidence: 99%

Online Social Network Profile Linkage

Zhang

Kan

Liu

et al. 2014

Information Retrieval Technology

View full text Add to dashboard Cite

Abstract. Piecing together social signals from people in different online social networks is key for downstream analytics. However, users may have different usernames in different social networks, making the linkage task difficult. To enable this, we explore a probabilistic approach that uses a domain-specific prior knowledge to address this problem of online social network user profile linkage. At scale, linkage approaches that are based on a naïve pairwise comparisons that have quadratic complexity become prohibitively expensive. Our proposed threshold-based canopying framework -named OPL -reduces this pairwise comparisons, and guarantees a upper bound theoretic linear complexity with respect to the dataset size. We evaluate our approaches on real-world, large-scale datasets obtained from Twitter and Linkedin. Our probabilistic classifier integrating prior knowledge into Naïve Bayes performs at over 85% F1-measure for pairwise linkage, comparable to state-of-the-art approaches.

show abstract

“…This field structure could be achieved by splitting unstructured/semi-structured addresses with address parsing. Moreover, there are hierarchical restrictions between these fields, which are useful to avoid unnecessary computation of field comparison [3,4]. These hierarchical restrictions can be mined from the semantic relationships between fields, which widely exist in real world record matching problems.…”

Section: Introductionmentioning

confidence: 99%

Improving Record Linkage Accuracy with Hierarchical Feature Level Information and Parsed Data

et al. 2017

View full text Add to dashboard Cite

Abstract. Probabilistic record linkage is a well established topic in the literature. Fellegi-Sunter probabilistic record linkage and its enhanced versions are commonly used methods, which calculate match and nonmatch weights for each pair of records. Bayesian network classifiersnaive Bayes classifier and TAN have also been successfully used here. Recently, an extended version of TAN (called ETAN) has been developed and proved superior in classification accuracy to conventional TAN. However, no previous work has applied ETAN to record linkage and investigated the benefits of using naturally existing hierarchical feature level information and parsed fields of the datasets. In this work, we extend the naive Bayes classifier with such hierarchical feature level information. Finally we illustrate the benefits of our method over previously proposed methods on 4 datasets in terms of the linkage performance (F1 score). We also show the results can be further improved by evaluating the benefit provided by additionally parsing the fields of these datasets.

show abstract

Efficient and Effective Duplicate Detection in Hierarchical Data

Cited by 28 publications

References 19 publications

Extending Naive Bayes Classifier with Hierarchy Feature Level Information for Record Linkage

Extending Naive Bayes Classifier with Hierarchy Feature Level Information for Record Linkage

Online Social Network Profile Linkage

Improving Record Linkage Accuracy with Hierarchical Feature Level Information and Parsed Data

Contact Info

Product

Resources

About