Reducing uncertainty of schema matching via crowdsourcing

Abstract

Schema matching is a central challenge for data integration systems. Automated tools are often uncertain about schema matchings they suggest, and this uncertainty is inherent since it arises from the inability of the schema to fully capture the semantics of the represented data. Human common sense can often help. Inspired by the popularity and the success of easily accessible crowdsourcing platforms, we explore the use of crowdsourcing to reduce the uncertainty of schema matching. Since it is typical to ask simple questions on crowdsourcing platforms, we assume that each question, namely Correspondence Correctness Question (CCQ), is to ask the crowd to decide whether a given correspondence should exist in the correct matching. We propose frameworks and efficient algorithms to dynamically manage the CCQs, in order to maximize the uncertainty reduction within a limited budget of questions. We develop two novel approaches, namely "Single CCQ" and "Multiple CCQ", which adaptively select, publish and manage the questions. We verified the value of our solutions with simulation and real implementation.

Pay-As-You-Go Multi-user Feedback Model for Ontology Matching

“…Multi-user scenarios include CrowdMap [19] for ontology matching, ZenCrowd [8] for entity linking, and Zhang et al [23] for database schema matching, which use crowdsourcing on a web platform.…”

Reconciling Schema Matching Networks Through Crowdsourcing

“…For example, in [22], the crowdsourcing is employed to validate the search results of automated image search on mobile devices. In [45], the authors leveraged the user CAPTCHAs inputs in web forms to recognize difficult words that cannot 14 Similar to our work, the authors of [46] also make use of crowdsourcing to validate the correspondences and reduce their uncertainty. However, they only focus on a pair-wise matching and using entropy-based decision strategy to maximize the uncertainty reduction at a single validation step.…”

A workload-dependent task assignment policy for crowdsourcing

“…Clearly, to be able to assign annotator j ∈ A to some task in her k-th iteration, she must have been assigned to some other k − 1 tasks in the past. We assume p jk to be known, as inferrable, e.g., from historical data or from the worker's performance on ad-hoc verification tasks, as, e.g., in [28,8]. Indeed, if we consider the perspective of a platform owner, it is reasonable to assume the availability of worker statistics for each task type supported by the platform, as done in [18].…”

“…However, differently from previous works, in this paper we study the problem of assigning annotators to tasks under the assumption that the annotators' reliability could change depending on their workload, as a result of, e.g., fatigue and learning. In this regard, a crowdsourcing platform may use all historic data and trends from previous sessions about its workers to provide a reliable estimate of the typical accuracy variations experienced by each annotator, as successfully done by several works in the literature [28,8,18]. These estimates allow us to devise task assignment policies whose attention to changes in annotators' accuracies proves extremely beneficial in terms of reduction of the classification error, as we will better illustrate in the remainder of the paper.…”