Crowdsourcing algorithms for entity resolution

Vesdapunt, Norases; Bellare, Kedar; Dalvi, Nilesh

doi:10.14778/2732977.2732982

Cited by 149 publications

(163 citation statements)

References 9 publications

Supporting

Mentioning

151

Contrasting

Order By: Relevance

“…[21,22,30,31]): -Amazon-Google 4 is a data set containing 4598 product records from 2 data sets. There are a total of 10,527,166 pairs of records, of which 1,300 represent the same entity.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Pay-as-you-go Configuration of Entity Resolution

Maskat

Paton

Embury

2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Entity resolution, which seeks to identify records that represent the same entity, is an important step in many data integration and data cleaning applications. However, entity resolution is challenging both in terms of scalability (all-against-all comparisons are computationally impractical) and result quality (syntactic evidence on record equivalence is often equivocal). As a result, end-to-end entity resolution proposals involve several stages, including blocking to efficiently identify candidate duplicates, detailed comparison to refine the conclusions from blocking, and clustering to identify the sets of records that may represent the same entity. However, the quality of the result is often crucially dependent on configuration parameters in all of these stages, for which it may be difficult for a human expert to provide suitable values. This paper describes an approach in which a complete entity resolution process is optimized, on the basis of feedback (such as might be obtained from crowds) on candidate duplicates. Given such feedback, an evolutionary search of the space of configuration parameters is carried out, with a view to maximizing the fitness of the resulting clusters. The approach is payas-you-go in that more feedback can be expected to give rise to better outcomes. An empirical evaluation shows that the co-optimization of the different stages in entity resolution can yield significant improvements over default parameters, even with small amounts of feedback.

show abstract

Section: Methodsmentioning

confidence: 99%

“…[16,30,32,33]). For example, in Isele et al, [16], feedback is sought using active learning on the record pairs on which candidate detailed comparison rules disagree the most.…”

Section: Related Workmentioning

confidence: 99%

Pay-as-you-go Configuration of Entity Resolution

Maskat

Paton

Embury

2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…We compare our ACD method against existing four state-of-theart algorithms: TransNode [44], TransM [47], CrowdER [46], and GCER [48]. Note that CrowdER does not specify the algorithm for clustering crowdsourced record pairs.…”

Section: Methodsmentioning

confidence: 99%

“…For ACD and PC-Pivot, we repeat each of them 5 times in each experiment and report the average measurements, since they are both randomized algorithms. Note that TransNode [44] does not incorporate any parallel mechanism to issue HITs in a batch manner. Therefore, we omit TransNode from the experiments on the number of crowd iterations.…”

Section: Methodsmentioning

confidence: 99%

“…There exist five crowd-based deduplication algorithms in the literature, namely, Crowdclustering [25], CrowdER [46], GCER [48], TransM [47], and a very recent approach by Vesdapunt et al [44] (referred to as TransNode). As we have explained CrowdER and TransM in Section 1, we focus on the other three approaches in the following.…”

Section: Crowd-based Deduplicationmentioning

confidence: 99%

See 1 more Smart Citation

Crowd-Based Deduplication

Wang

Xiao

Lee

2015

Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Data deduplication stands as a building block for data integration and data cleaning. The state-of-the-art techniques focus on how to exploit crowdsourcing to improve the accuracy of deduplication. However, they either incur significant overheads on the crowd or offer inferior accuracy. This paper presents ACD, a new crowd-based algorithm for data deduplication. The basic idea of ACD is to adopt correlation clustering (which is a classic machine-based algorithm for data deduplication) under a crowd-based setting. We propose non-trivial techniques to reduce the time required in performing correlation clustering with the crowd, and devise methods to postprocess the results of correlation clustering for better accuracy of deduplication. With extensive experiments on the Amazon Mechanical Turk, we demonstrate that ACD outperforms the states of the art by offering a high precision of deduplication while incurring moderate crowdsourcing overheads.

show abstract

Active Learning Based Similarity Filtering for Efficient and Effective Record Linkage

Nanayakkara

Christen

Ranbaduge

2021

Advances in Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

The limited analytical value of using individual databases on their own increasingly requires the integration of large and complex databases for advanced data analytics. Linking personal medical records with travel and immigration data, for example, will allow the effective management of pandemics such as the current COVID-19 outbreak by tracking potentially infected individuals and their contacts. One major challenge for accurate linkage of large databases is the quadratic or even higher computational complexities of many advanced linkage algorithms. In this paper we present a novel approach that, based on the expected number of true matches between two databases, applies active learning to remove compared record pairs that are likely non-matches before a computationally expensive classification or clustering algorithm is employed to classify record pairs. Unlike blocking and indexing techniques that are used to reduce the number of record pairs to be compared, using recursive binning on a data dimension such as time or space, our approach removes likely non-matching record pairs in each bin after their comparison. Experiments on two real-world databases show that similarity filtering can substantially reduce run time and improve precision, at the costs of a small reduction in recall, of the final linkage results.

show abstract

Crowdsourcing algorithms for entity resolution

Cited by 149 publications

References 9 publications

Pay-as-you-go Configuration of Entity Resolution

Pay-as-you-go Configuration of Entity Resolution

Crowd-Based Deduplication

Active Learning Based Similarity Filtering for Efficient and Effective Record Linkage

Contact Info

Product

Resources

About