Ambiguity Helps: Classification with Disagreements in Crowdsourced Annotations

Sharmanska, Viktoriia; Hernández-Lobato, Daniel; Hernández-Lobato, José Miguel; Quadrianto, Novi

doi:10.1109/cvpr.2016.241

Cited by 23 publications

(24 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The white noise kernel is expressed as k(x x x, x x x ) = r if x x x = x x x and 0 otherwise, where r is the noise level. In the future, we will look into ways of dealing with the problem of input-dependent noise level (i.e., different degree of human mistakes and disagreements for different design instances) [38,39].…”

Section: Novelty Discoverymentioning

confidence: 99%

Beyond the Known: Detecting Novel Feasible Domains Over an Unbounded Design Space

Chen

Fuge

2017

Journal of Mechanical Design

View full text Add to dashboard Cite

To solve a design problem, sometimes it is necessary to identify the feasible design space. For design spaces with implicit constraints, sampling methods are usually used. These methods typically bound the design space; that is, limit the range of design variables. But bounds that are too small will fail to cover all possible designs; while bounds that are too large will waste sampling budget. This paper tries to solve the problem of efficiently discovering (possibly disconnected) feasible domains in an unbounded design space. We propose a data-driven adaptive sampling technique-ε-margin sampling, which both learns the domain boundary of feasible designs, while also expanding our knowledge of the design space as available budget increases. This technique is dataefficient, in that it makes principled probabilistic trade-offs between refining existing domain boundaries versus expanding the design space.We demonstrate that this method can better identify feasible domains on standard test functions compared to both random and active sampling (via uncertainty sampling). However, a fundamental problem when applying adaptive sampling to real world designs is that designs often have high dimensionality and thus require (in the worst case) exponentially more samples per dimension. We show how coupling Design Manifolds with ε-margin * Corresponding author.sampling allows us to actively expand high-dimensional design spaces without incurring this exponential penalty.We demonstrate this on real-world examples of glassware and bottle design, where our method discovers designs that have different appearance and functionality from its initial design set.

show abstract

Section: Novelty Discoverymentioning

confidence: 99%

Beyond the Known: Detecting Novel Feasible Domains Over an Unbounded Design Space

Chen

Fuge

2017

Journal of Mechanical Design

View full text Add to dashboard Cite

show abstract

“…Ann ( Paun et al, 2018;Pavlick and Kwiatkowski, 2019) and computer vision (Sharmanska et al, 2016) has leveraged annotator uncertainty to improve modeling. Thus, for our setting, we ask the following research question: RQ1: Is there inherent ambiguity in the language that expresses economic policy uncertainty?…”

Section: Subsetmentioning

confidence: 99%

Uncertainty over Uncertainty: Investigating the Assumptions, Annotations, and Text Measurements of Economic Policy Uncertainty

Keith

Teichmann

O’Connor

et al. 2020

Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science

View full text Add to dashboard Cite

Methods and applications are inextricably linked in science, and in particular in the domain of text-as-data. In this paper, we examine one such text-as-data application, an established economic index that measures economic policy uncertainty from keyword occurrences in news. This index, which is shown to correlate with firm investment, employment, and excess market returns, has had substantive impact in both the private sector and academia. Yet, as we revisit and extend the original authors' annotations and text measurements we find interesting text-as-data methodological research questions: (1) Are annotator disagreements a reflection of ambiguity in language?(2) Do alternative text measurements correlate with one another and with measures of external predictive validity? We find for this application (1) some annotator disagreements of economic policy uncertainty can be attributed to ambiguity in language, and (2) switching measurements from keyword-matching to supervised machine learning classifiers results in low correlation, a concerning implication for the validity of the index.

show abstract

“…Many researchers have concluded that rather than attempting to eliminate disagreements from annotated corpora, we should preserve them-indeed, some researchers have argued that corpora should aim to collect all distinct interpretations of an expression (Smyth et al, 1994;Poesio and Artstein, 2005;Aroyo and Welty, 2015;Sharmanska et al, 2016;Plank, 2016;Kenyon-Dean et al, 2018;Firman et al, 2018;Pavlick and Kwiatkowski, 2019). Poesio and Artstein (2005) and Recasens et al (2012) suggest that the best way to create resources capturing disagreements is by preserving implicit ambiguity-i.e., having multiple annotators label the items, and then keeping all these annotations, not just an aggregated 'gold standard'.…”

Section: Introductionmentioning

confidence: 99%

SemEval-2021 Task 12: Learning with Disagreements

Uma¹,

Fornaciari²,

Dumitrache³

et al. 2021

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)

View full text Add to dashboard Cite

Disagreement between coders is ubiquitous in virtually all datasets annotated with human judgements in both natural language processing and computer vision. However, most supervised machine learning methods assume that a single preferred interpretation exists for each item, which is at best an idealization. The aim of the SemEval-2021 shared task on Learning with Disagreements (L --D ) was to provide a unified testing framework for methods for learning from data containing multiple and possibly contradictory annotations covering the best-known datasets containing information about disagreements for interpreting language and classifying images. In this paper we describe the shared task and its results.

show abstract

Ambiguity Helps: Classification with Disagreements in Crowdsourced Annotations

Abstract: Abstract

Cited by 23 publications

References 23 publications

Beyond the Known: Detecting Novel Feasible Domains Over an Unbounded Design Space

Beyond the Known: Detecting Novel Feasible Domains Over an Unbounded Design Space

Uncertainty over Uncertainty: Investigating the Assumptions, Annotations, and Text Measurements of Economic Policy Uncertainty

SemEval-2021 Task 12: Learning with Disagreements

Contact Info

Product

Resources

About