Attention-Aware Answers of the Crowd

Tu, Jingzheng; Yu, Guoxian; Wang, Jun; Domeniconi, Carlotta; Zhang, Xiangliang

doi:10.1137/1.9781611976236.51

Cited by 8 publications

(3 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This process is human-resource intensive and requires expertise in the subject area, in which the annotation takes place. In recent years, crowdsourcing has been used to alleviate the demand of elaborate human expertise [39].…”

Section: Discussion Of the Findingsmentioning

confidence: 99%

Assessing the difficulty of annotating medical data in crowdworking with help of experiments

et al. 2021

View full text Add to dashboard Cite

Background As healthcare-related data proliferate, there is need to annotate them expertly for the purposes of personalized medicine. Crowdworking is an alternative to expensive expert labour. Annotation corresponds to diagnosis, so comparing unlabeled records to labeled ones seems more appropriate for crowdworkers without medical expertise. We modeled the comparison of a record to two other records as a triplet annotation task, and we conducted an experiment to investigate to what extend sensor-measured stress, task duration, uncertainty of the annotators and agreement among the annotators could predict annotation correctness. Materials and methods We conducted an annotation experiment on health data from a population-based study. The triplet annotation task was to decide whether an individual was more similar to a healthy one or to one with a given disorder. We used hepatic steatosis as example disorder, and described the individuals with 10 pre-selected characteristics related to this disorder. We recorded task duration, electro-dermal activity as stress indicator, and uncertainty as stated by the experiment participants (n = 29 non-experts and three experts) for 30 triplets. We built an Artificial Similarity-Based Annotator (ASBA) and compared its correctness and uncertainty to that of the experiment participants. Results We found no correlation between correctness and either of stated uncertainty, stress and task duration. Annotator agreement has not been predictive either. Notably, for some tasks, annotators agreed unanimously on an incorrect annotation. When controlling for Triplet ID, we identified significant correlations, indicating that correctness, stress levels and annotation duration depend on the task itself. Average correctness among the experiment participants was slightly lower than achieved by ASBA. Triplet annotation turned to be similarly difficult for experts as for non-experts. Conclusion Our lab experiment indicates that the task of triplet annotation must be prepared cautiously if delegated to crowdworkers. Neither certainty nor agreement among annotators should be assumed to imply correct annotation, because annotators may misjudge difficult tasks as easy and agree on incorrect annotations. Further research is needed to improve visualizations for complex tasks, to judiciously decide how much information to provide, Out-of-the-lab experiments in crowdworker setting are needed to identify appropriate designs of a human-annotation task, and to assess under what circumstances non-human annotation should be preferred.

show abstract

Section: Discussion Of the Findingsmentioning

confidence: 99%

Assessing the difficulty of annotating medical data in crowdworking with help of experiments

et al. 2021

View full text Add to dashboard Cite

show abstract

“…General user attention covers the overall attention a user gives to an user interface without considering the task-specific information. Studies have shown that general user attention improves human performance in different work and play activities [45], [46]. We hypothesize that crowdsourcing is another workrelated activity that demands user attention for carrying out the tasks well.…”

Section: Research Objectivesmentioning

confidence: 96%

Your Cursor Reveals: On Analyzing Workers’ Browsing Behavior and Annotation Quality In Crowdsourcing Tasks

Lo¹,

Lim²

2024

IEEE Access

View full text Add to dashboard Cite

In this work, we investigate the connection between browsing behavior and task quality of crowdsourcing workers performing annotation tasks that require information judgements. Such information judgements are often required to derive ground truth answers to information retrieval queries. We explore the use of workers' browsing behavior to directly determine their annotation result quality. We hypothesize user attention to be the main factor contributing to a worker's annotation quality. To predict annotation quality at the task level, we model two aspects of task-specific user attention, also known as general and semantic user attentions. Both aspects of user attention can be modeled using different types of browsing behavior features but most previous research mostly focuses on the former. This work therefore proposes to model semantic user attention by capturing the worker's understanding of task content using task-semantics specific behavior features. We develop a web-based annotation interface for gathering user behavior data when workers perform a knowledge path retrieval task. With the collected data, we train several prediction models using behavior features corresponding to different aspects of user attention and conduct experiments on a set of annotation tasks performed by 51 Amazon Mechanical Turk workers. We show that the prediction model using both general and semantic user attention features can achieve the best performance of nearly 75% accuracy.

show abstract

“…CrowdDQS dynamically issues golden standard questions and estimate the accuracies of workers in real-time, then it can select workers with higher accuracies for task assignment [14]. Tu et al suggest that the attention of workers changes over time, thus the accuracy of workers can not be kept constant, therefore, they proposed a probabilistic model that takes into account workers' attention [38]. Compared to these models, this paper focuses on a different attention weights in COR scenario and our assumption is that the ability of a worker is diverse but constant (i.e.…”

Section: Crowdsourcingmentioning

confidence: 99%

Exploiting Heterogeneous Graph Neural Networks with Latent Worker/Task Correlation Information for Label Aggregation in Crowdsourcing

Wu¹,

Ma²,

Wu³

et al. 2020

Preprint

View full text Add to dashboard Cite

Crowdsourcing has attracted much attention for its convenience to collect labels from non-expert workers instead of experts. However, due to the high level of noise from the non-experts, an aggregation model that learns the true label by incorporating the source credibility is required. In this paper, we propose a novel framework based on graph neural networks for aggregating crowd labels. We construct a heterogeneous graph between workers and tasks and derive a new graph neural network to learn the representations of nodes and the true labels. Besides, we exploit the unknown latent interaction between the same type of nodes (workers or tasks) by adding a homogeneous attention layer in the graph neural networks. Experimental results on 13 real-world datasets show superior performance over state-of-the-art models.

show abstract

Attention-Aware Answers of the Crowd

Cited by 8 publications

References 24 publications

Assessing the difficulty of annotating medical data in crowdworking with help of experiments

Assessing the difficulty of annotating medical data in crowdworking with help of experiments

Your Cursor Reveals: On Analyzing Workers’ Browsing Behavior and Annotation Quality In Crowdsourcing Tasks

Exploiting Heterogeneous Graph Neural Networks with Latent Worker/Task Correlation Information for Label Aggregation in Crowdsourcing

Contact Info

Product

Resources

About