Effective Crowdsourcing for a New Type of Summarization Task

Jiang, Youxuan; Finegan-Dollak, Catherine; Kummerfeld, Jonathan K.; Lasecki, Walter S.

doi:10.18653/v1/n18-2099

Cited by 15 publications

(6 citation statements)

References 20 publications

(10 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additional iterations of this procedure are likely to improve microaggression recognition and substantially increase the seize of the corpus. We note that one option is to have workers generate examples, rather than rate (e.g., Xu et al, 2013;Su et al, 2016;Jiang et al, 2018); however, such a process raises ethical concerns of having crowdworkers generate toxic statements towards others. Second, our current focus is on gender-based MAS.…”

Section: Discussion and Limitationsmentioning

confidence: 99%

Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts

Breitfeller¹,

Ahn²,

Jurgens

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

104

View full text Add to dashboard Cite

Microaggressions are subtle, often veiled, manifestations of human biases. These uncivil interactions can have a powerful negative impact on people by marginalizing minorities and disadvantaged groups. The linguistic subtlety of microaggressions in communication has made it difficult for researchers to analyze their exact nature, and to quantify and extract microaggressions automatically. Specifically, the lack of a corpus of real-world microaggressions and well-defined criteria for annotating them have prevented researchers from addressing these problems at scale. In this paper, we devise a general but nuanced, computationally operationalizable typology of microaggressions based on a small subset of microaggression data that we have. We then create two datasets: one with examples of diverse types of microaggressions recollected by their targets, and another with gender-based microaggressions in public conversations on social media. We introduce a new, more objective criterion for annotation and an activelearning based procedure that increases the likelihood of surfacing posts containing microaggressions. Finally, we analyze the trends that emerge from these new datasets.

show abstract

Section: Discussion and Limitationsmentioning

confidence: 99%

Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts

Breitfeller¹,

Ahn²,

Jurgens

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

104

View full text Add to dashboard Cite

show abstract

“…Some researchers have used crowdsourcing platforms to create new language resources (Roit et al, 2020;Jiang et al, 2018). Some work created a parallel corpus for domain-adaptation by asking crowdworkers to translate in-domain monolingual sentences (Zaidan and Callison-Burch, 2011;Behnke et al, 2018;Kalimuthu et al, 2019).…”

Section: Creating Parallel Corpus With Crowdworkersmentioning

confidence: 99%

NTT Neural Machine Translation Systems at WAT 2019

Morishita

Suzuki

Nagata

2019

Proceedings of the 6th Workshop on Asian Translation

View full text Add to dashboard Cite

In this paper, we describe our systems that were submitted to the translation shared tasks at WAT 2019. This year, we participated in two distinct types of subtasks, a scientific paper subtask and a timely disclosure subtask, where we only considered English-to-Japanese and Japanese-to-English translation directions. We submitted two systems (En-Ja and Ja-En) for the scientific paper subtask and two systems (Ja-En, texts, items) for the timely disclosure subtask. Three of our four systems obtained the best human evaluation performances. We also confirmed that our new additional web-crawled parallel corpus improves the performance in unconstrained settings.

show abstract

“…Given numerous prior work noting that summaries written by crowd workers exhibit limitations associated with lack of annotator expertise in the domain (Gillick and Liu, 2010), especially at narrower tasks like query-based summarization (Jiang et al, 2018), we use LLM-generated references for benchmarking reference-based methods instead.…”

Section: Reference-based Methodsmentioning

confidence: 99%

Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)

2023

View full text Add to dashboard Cite

Language models (LMs) have been argued to overlap substantially with human beings in grammaticality judgment tasks. But when humans systematically make errors in language processing, should we expect LMs to behave like cognitive models of language and mimic human behavior? We answer this question by investigating LMs' more subtle judgments associated with "language illusions" -sentences that are vague in meaning, implausible, or ungrammatical but receive unexpectedly high acceptability judgments by humans. We looked at three illusions: the comparative illusion (e.g. "More people have been to Russia than I have"), the depth-charge illusion (e.g. "No head injury is too trivial to be ignored"), and the negative polarity item (NPI) illusion (e.g. "The hunter who no villager believed to be trustworthy will ever shoot a bear"). We found that probabilities represented by LMs were more likely to align with human judgments of being "tricked" by the NPI illusion which examines a structural dependency, compared to the comparative and the depth-charge illusions which require sophisticated semantic understanding. No single LM or metric yielded results that are entirely consistent with human behavior. Ultimately, we show that LMs are limited both in their construal as cognitive models of human language processing and in their capacity to recognize nuanced but critical information in complicated language materials.

show abstract

Effective Crowdsourcing for a New Type of Summarization Task

Cited by 15 publications

References 20 publications

Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts

Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts

NTT Neural Machine Translation Systems at WAT 2019

Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)

Contact Info

Product

Resources

About