Robust and efficient fuzzy match for online data cleaning

Chaudhuri, Surajit; Ganjam, Kris; Ganti, Venkatesh; Motwani, Rajeev

doi:10.1145/872757.872796

Cited by 343 publications

(287 citation statements)

References 12 publications

Supporting

Mentioning

276

Contrasting

Unclassified

Order By: Relevance

“…An alternative to standard blocking is the sorted neighbourhood [24] approach, where records are sorted according to the values of the blocking variable, then a sliding window is moved over the sorted records, and comparisons are performed between the records within the window. Newer experimental approaches based on approximate q-gram indices [2,7] or high-dimensional clustering [29] are current research topics. The effects of blocking upon the quality and complexity of the data linkage process are discussed in more details in Section 5.…”

Section: Data Linkage Processmentioning

confidence: 99%

“…Their results on various data sets show that learned edit distance resulted in improved precision and recall results. Very similar approaches are presented in [7,30,46,47], with [30] using support vector machines for the binary classification task of record pairs. As shown in [12], combining different learned string comparison methods can result in improved linkage classification.…”

Section: Modern Approachesmentioning

confidence: 99%

See 1 more Smart Citation

Quality and Complexity Measures for Data Linkage and Deduplication

Christen

Goiser

2007

Studies in Computational Intelligence

151

126

View full text Add to dashboard Cite

Abstract. Deduplicating one data set or linking several data sets are increasingly important tasks in the data preparation steps of many data mining projects. The aim is to match all records relating to the same entity. Research interest in this area has increased in recent years, with techniques originating from statistics, machine learning, information retrieval, and database research being combined and applied to improve the linkage quality, as well as to increase performance and efficiency when linking or deduplicating very large data sets. Different measures have been used to characterise the quality and complexity of data linkage algorithms, and several new metrics have been proposed. An overview of the issues involved in measuring data linkage and deduplication quality and complexity is presented in this chapter. It is shown that measures in the space of record pair comparisons can produce deceptive accuracy results. Various measures are discussed and recommendations are given on how to assess data linkage and deduplication quality and complexity.

show abstract

Section: Data Linkage Processmentioning

confidence: 99%

Section: Modern Approachesmentioning

confidence: 99%

Quality and Complexity Measures for Data Linkage and Deduplication

Christen

Goiser

2007

Studies in Computational Intelligence

151

126

View full text Add to dashboard Cite

show abstract

“…Recognizing the importance of this problem, a variety of commercial products (see, e.g., [1]) and research prototypes (see, e.g., [21]) target the space of data cleaning, offering an array of techniques to identify and correct data quality problems. At a high level, data cleaning solutions can be classified into two broad categories: (a) those that operate on top of an RDBMS using an SQL interface to express and realize data cleaning tasks [16,17,18,7] and (b) those that extract the relevant data out of a database and operate on them using proprietary techniques and interfaces [1].…”

Section: Permission To Copy Without Fee All or Part Of This Materials mentioning

confidence: 99%

“…which abound in customer related databases. These include techniques for indexed retrieval of strings based on notions of approximate string match [7], correlating string attributes using string similarity predicates (e.g., cosine similarity, edit distance and variants thereof) [17,18,6,5] and deploying algorithms and/or rule engines for automatically correcting/transforming strings into canonical forms [1, 3,22]. These techniques will successfully match a query string (or a collection of strings) approximately (for suitably defined notions of approximate match) against the values of an attribute in a relation R. For a given query string, such techniques can tag each matching attribute value with a score quantifying the degree of similarity (closeness) of the query string to the attribute value string.…”

Section: Permission To Copy Without Fee All or Part Of This Materials mentioning

confidence: 99%

See 1 more Smart Citation

Merging the Results of Approximate Match Operations

Guha¹,

Koudas

Marathe

et al. 2004

Proceedings 2004 VLDB Conference

View full text Add to dashboard Cite

The development of a mobile app‐focused deduplication strategy for the Apple Heart Study that informs recommendations for future digital trials

García

Lee

Balasubramanian

et al. 2022

Stat

View full text Add to dashboard Cite

An app‐based clinical trial enrolment process can contribute to duplicated records, carrying data management implications. Our objective was to identify duplicated records in real time in the Apple Heart Study (AHS). We leveraged personal identifiable information (PII) to develop a dissimilarity score (DS) using the Damerau–Levenshtein distance. For computational efficiency, we focused on four types of records at the highest risk of duplication. We used the receiver operating curve (ROC) and resampling methods to derive and validate a decision rule to classify duplicated records. We identified 16,398 (4%) duplicated participants, resulting in 419,297 unique participants out of a total of 438,435 possible. Our decision rule yielded a high positive predictive value (96%) with negligible impact on the trial's original findings. Our findings provide principled solutions for future digital trials. When establishing deduplication procedures for digital trials, we recommend collecting device identifiers in addition to participant identifiers; collecting and ensuring secure access to PII; conducting a pilot study to identify reasons for duplicated records; establishing an initial deduplication algorithm that can be refined; creating a data quality plan that informs refinement; and embedding the initial deduplication algorithm in the enrolment platform to ensure unique enrolment and linkage to previous records.

show abstract

Robust and efficient fuzzy match for online data cleaning

Cited by 343 publications

References 12 publications

Quality and Complexity Measures for Data Linkage and Deduplication

Quality and Complexity Measures for Data Linkage and Deduplication

Merging the Results of Approximate Match Operations

The development of a mobile app‐focused deduplication strategy for the Apple Heart Study that informs recommendations for future digital trials

Contact Info

Product

Resources

About