A Cognitive Model of Human Bias in Matching

PoWareMatch: a Quality-aware Deep Learning Approach to Improve Human Schema Matching

“…The theoretical setting described above requires human matchers to offer unbiased matching (which we formally define in this paper), assuming human matchers are experts in matching. This is, unfortunately, not always the case and human matchers were shown to have cognitive biases when matching [9], which may lead to poor decision making. Rather than aggregately judge human matcher proficiency and discard those that may seem to provide inferior decisions, we propose to directly overcome human biases and accept only high-quality decisions.…”

“…Human schema matching has been recently analyzed using metacognitive psychology [9], a discipline that investigates factors impacting humans when performing knowledge intensive tasks [11]. The metacognitive approach [16], traditionally applied for learning and answering knowledge questions, highlights the role of subjective confidence in regulating efforts while performing tasks.…”

“…Therefore, human subjective confidence is unlikely to serve as a (single) good predictor to matching correctness. Ackerman et al reported several possible biases in human matching which affect their confidence and their ability to provide accurate matching [9]. The interested reader to Appendix B for additional details.…”

“…A feature encoding of a human matcher decision ℎ 𝑡 uses a 4-dimensional feature vector, composed of the reported confidence, allocated time, consensual agreement, and an algorithmic similarity score. Allocated time and consensual agreement (along with control, the extent of which the matcher was assisted by an algorithmic solution, which we do not use here since it was shown to be less predictive in our experiments) are human biases studied by Ackerman et al [9] (see Appendix B for more details). Allocated time is measured directly using history timestamps.…”

“…So far, challenges raised by human matching were answered by pulling further on human resources, using crowdsourcing (e.g., [26,48,54,61,62]) and pay-as-you-go frameworks (e.g., [42,47,51]). However, recent research has challenged both traditional and new methods for human-in-the-loop matching, showing that humans have cognitive biases that decrease their ability to perform matching tasks effectively [9]. For example, the study shows that over time, human matchers are willing to determine that an element pair matches despite their low confidence in the match, possibly leading to poor performance.…”