Goldilocks: Consistent Crowdsourced Scalar Annotations with Relative Uncertainty

Abstract

Human ratings have become a crucial resource for training and evaluating machine learning systems. However, traditional elicitation methods for absolute and comparative rating suffer from issues with consistency and often do not distinguish between uncertainty due to disagreement between annotators and ambiguity inherent to the item being rated. In this work, we present Goldilocks, a novel crowd rating elicitation technique for collecting calibrated scalar annotations that also distinguishes inherent ambiguity from inter-annotator disagreement. We introduce two main ideas: grounding absolute rating scales with examples and using a two-step bounding process to establish a range for an item's placement. We test our designs in three domains: judging toxicity of online comments, estimating satiety of food depicted in images, and estimating age based on portraits. We show that (1) Goldilocks can improve consistency in domains where interpretation of the scale is not universal, and that (2) representing items with ranges lets us simultaneously capture different sources of uncertainty leading to better estimates of pairwise relationship distributions.

Confidence Contours: Uncertainty-Aware Annotation for Medical Semantic Segmentation

“…Some researchers propose the use of rationales as answers (Donahue and Grauman 2011;McDonnell et al 2016), while others propose open-ended answers that are then clustered or taxonomized (Kairam and Heer 2016;Chang, Amershi, and Kamar 2017). Finally, some have proposed more middle ground solutions in the form of new annotation representations, such as ranges in scalar rating annotation (Chen, Weld, and Zhang 2021). Instead of asking for confidence or uncertainty via a question separate from the annotation itself, range annotations enable annotators to directly convey uncertainty calibrated to their annotation.…”

Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems

“…3.5.2 Providing insights from past annotation decisions. Exploring different strategies for better grasping the importance of elements in the annotation task, possibly by relying on the collective wisdom or opinions of others, has been useful in past literature[17,18]. P4 mentions sharing what other people thought or getting insights from others' perspectives might be more helpful in determining the significance of certain aspects of the task, as he remarked, "This was relevant because of this aspect or something like that.…”