Learning to Characterize Matching Experts

Abstract

Matching is a task at the heart of any data integration process, aimed at identifying correspondences among data elements. Matching problems were traditionally solved in a semi-automatic manner, with correspondences being generated by matching algorithms and outcomes subsequently validated by human experts. Human-in-the-loop data integration has been recently challenged by the introduction of big data and recent studies have analyzed obstacles to effective human matching and validation. In this work we characterize human matching experts, those humans whose proposed correspondences can mostly be trusted to be valid. We provide a novel framework for characterizing matching experts that, accompanied with a novel set of features, can be used to identify reliable and valuable human experts. We demonstrate the usefulness of our approach using an extensive empirical evaluation. In particular, we show that our approach can improve matching results by filtering out inexpert matchers.

HumanAL: Calibrating Human Matching Beyond a Single Task

“…(2) Majority decision and the user decision (Figure 2) are the most significant features. (3) Different from the observations in the setting of open-ended crowdsourced matching [3,44,45], decision times are less significant in boolean crowdsourcing.…”

“…Recently, Ackerman et al detected human cognitive biases affecting matching quality [3]. These biases were used in the scope of open-ended crowdsourcing [36] for the task of improving human schema matching [44,45]. The current work focuses on boolean crowdsourcing aiming to address multiple human matching tasks.…”

“…Consequently, human-in-the-loop solutions [14,26,27,29,32,34,37,41,58], such as crowdsourcing, were developed to enable cheaper acquisition of labels from a large number of annotators in a short time. However, the increasing availability of human-labeled data may also yield variation in the reliability and proficiency of these human annotators [3,12,39,44,46,56], which, in turn, impair the quality of the generated labels. This variation may arise due to different expertise levels [12,46,56], mood changes [43], various cognitive biases [3,44] and more.…”

“…However, the increasing availability of human-labeled data may also yield variation in the reliability and proficiency of these human annotators [3,12,39,44,46,56], which, in turn, impair the quality of the generated labels. This variation may arise due to different expertise levels [12,46,56], mood changes [43], various cognitive biases [3,44] and more.…”

“…Specifically, we perform an empirical study with the structurally rich database task of schema matching (SM) [5,38], the structured task of entity matching (EM) [7,13,16] and the textually rich NLP task of labeling semantically equivalent sentences (text matching, TM) [19,35]. SM is characterized as a very challenging task [44] and was shown in literature to require expert skills [12,44,56]. EM usually depends on context and may be confusing for humans (e.g., what it means to be a match) [10,11].…”