InCognitoMatch: Cognitive-aware Matching via Crowdsourcing

Shraga, Roee; Scharf, Coral; Ackerman, Rakefet; Gal, Avigdor

doi:10.1145/3318464.3384697

Cited by 10 publications

(8 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In future work, we aim to extend quality assessment to handle varying scales and platforms. Specifically, a common way to assess a subset of the problem is by using crowdsourcing (e.g., using [42]), where several additional aspects, such as the heterogeneity of crowd workers [35], need to be considered. Additionally, we aim to explore more facets of behavioral change in the context of crowdsourcing.…”

Section: Discussionmentioning

confidence: 99%

Learning to Characterize Matching Experts

Shraga

Amir

Gal

2021

2021 IEEE 37th International Conference on Data Engineering (ICDE)

Self Cite

View full text Add to dashboard Cite

Matching is a task at the heart of any data integration process, aimed at identifying correspondences among data elements. Matching problems were traditionally solved in a semi-automatic manner, with correspondences being generated by matching algorithms and outcomes subsequently validated by human experts. Human-in-the-loop data integration has been recently challenged by the introduction of big data and recent studies have analyzed obstacles to effective human matching and validation. In this work we characterize human matching experts, those humans whose proposed correspondences can mostly be trusted to be valid. We provide a novel framework for characterizing matching experts that, accompanied with a novel set of features, can be used to identify reliable and valuable human experts. We demonstrate the usefulness of our approach using an extensive empirical evaluation. In particular, we show that our approach can improve matching results by filtering out inexpert matchers.

show abstract

Section: Discussionmentioning

confidence: 99%

Learning to Characterize Matching Experts

Shraga

Amir

Gal

2021

2021 IEEE 37th International Conference on Data Engineering (ICDE)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In our case, the model would assign non-zero weights to 23 input features. However, such weights are too noisy to be of use and are quite overwhelming for human understanding (the human cognitive load is usually limited to understanding around 7-10 features at a time (Shraga et al, 2020)).…”

Section: Bootstrap With Predictive Modelsmentioning

confidence: 99%

Real-time wildfire detection with semantic explanations

Phan¹,

Quach²,

Nguyen³

et al. 2022

Expert Systems with Applications

View full text Add to dashboard Cite

“…The overall time and (internal) normalized decision time with respect to other decisions made by the user are internal global features. This set also includes a prior intelligence estimation using psychometric riddles [4,48].…”

Section: From Human Behavior To Feature Extraction To Calibrationmentioning

confidence: 99%

“…4.1.1 Interface. We adopt the interface of IncognitoMatch [48], which was originally designed for schema matching problems. IncognitoMatch is a crowdsourcing platform that also gathers cognitive information such as decision times.…”

Section: Experimental Settingsmentioning

confidence: 99%

See 1 more Smart Citation

HumanAL: Calibrating Human Matching Beyond a Single Task

Shraga¹

2022

Preprint

Self Cite

View full text Add to dashboard Cite

This work offers a novel view on the use of human input as labels, acknowledging that humans may err. We build a behavioral profile for human annotators which is used as a feature representation of the provided input. We show that by utilizing black-box machine learning, we can take into account human behavior and calibrate their input to improve the labeling quality. To support our claims and provide a proof-of-concept, we experiment with three different matching tasks, namely, schema matching, entity matching and text matching. Our empirical evaluation suggests that the method can improve the quality of gathered labels in multiple settings including cross-domain (across different matching tasks).

show abstract

InCognitoMatch: Cognitive-aware Matching via Crowdsourcing

Cited by 10 publications

References 4 publications

Learning to Characterize Matching Experts

Learning to Characterize Matching Experts

Real-time wildfire detection with semantic explanations

HumanAL: Calibrating Human Matching Beyond a Single Task

Contact Info

Product

Resources

About