Cross-language transfer of semantic annotation via targeted crowdsourcing

Chowdhury, Shammur Absar; Ghosh, Arindam; Stepanov, Evgeny A.; Bayer, Ali Orkan; Riccardi, Giuseppe; Klasinas, Ioannis

doi:10.21437/interspeech.2014-478

Cited by 8 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…; L11: Sympathy and support: Tweets with prayers, thoughts, and emotional support; For the manual annotation, we opted to use Amazon Mechanical Turk (AMT) platform. In crowdsourcing, one of the challenges is to find a large number of qualified workers while filtering out low-quality workers or spammers (Chowdhury et al 2014). To tackle this problem, a typical approach is to use qualification tests followed by a gold standard evaluation (Chowdhury et al 2015).…”

Section: Manual Annotationsmentioning

confidence: 99%

HumAID: Human-Annotated Disaster Incidents Data from Twitter with Deep Learning Benchmarks

Alam

Qazi

Imran

et al. 2021

ICWSM

View full text Add to dashboard Cite

Social networks are widely used for information consumption and dissemination, especially during time-critical events such as natural disasters. Despite its significantly large volume, social media content is often too noisy for direct use in any application. Therefore, it is important to filter, categorize, and concisely summarize the available content to facilitate effective consumption and decision-making. To address such issues automatic classification systems have been developed using supervised modeling approaches, thanks to the earlier efforts on creating labeled datasets. However, existing datasets are limited in different aspects (e.g., size, contains duplicates) and less suitable to support more advanced and data-hungry deep learning models. In this paper, we present a new large-scale dataset with ~77K human-labeled tweets, sampled from a pool of ~24 million tweets across 19 disaster events that happened between 2016 and 2019. Moreover, we propose a data collection and sampling pipeline, which is important for social media data sampling for human annotation. We report multiclass classification results using classic and deep learning (fastText and transformer) based models to set the ground for future studies. The dataset and associated resources are publicly available at https://crisisnlp.qcri.org/humaid_dataset.html.

show abstract

Section: Manual Annotationsmentioning

confidence: 99%

HumAID: Human-Annotated Disaster Incidents Data from Twitter with Deep Learning Benchmarks

Alam

Qazi

Imran

et al. 2021

ICWSM

View full text Add to dashboard Cite

show abstract

“…If we compare our task with other previous tasks that used a similar metric, we can better understand the complexity of the task and judge the quality of the annotations. Chowdhury et al (2014) worked on the task of semantic annotations of the utterances from conversations. One of the sub-task annotators had to perform was selecting a text span describing a hardware concept.…”

Section: Discussionmentioning

confidence: 99%

“…An alternative agreement measure that does not require the knowledge of true negatives for its calculations is Positive (Specific) Agreement (Fleiss, 1975), which is similar to the widely used F-measure (Hripcsak and Rothschild, 2005). It has previously been shown to be useful in the evaluation of crowdsourced annotations tasks, similar to our's (Stepanov et al, 2018;Chowdhury et al, 2014).…”

Section: Evaluation Of Annotationsmentioning

confidence: 99%

Annotation of Emotion Carriers in Personal Narratives

Tammewar,

Cervone,

Messner

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We are interested in the problem of understanding personal narratives (PN) -spoken or written -recollections of facts, events, and thoughts. In PN, emotion carriers are the speech or text segments that best explain the emotional state of the user. Such segments may include entities, verb or noun phrases. Advanced automatic understanding of PNs requires not only the prediction of the user emotional state but also to identify which events (e.g. the loss of relative or the visit of grandpa) or people ( e.g. the old group of high school mates) carry the emotion manifested during the personal recollection. This work proposes and evaluates an annotation model for identifying emotion carriers in spoken personal narratives. Compared to other text genres such as news and microblogs, spoken PNs are particularly challenging because a narrative is usually unstructured, involving multiple sub-events and characters as well as thoughts and associated emotions perceived by the narrator. In this work, we experiment with annotating emotion carriers from speech transcriptions in the Ulm State-of-Mind in Speech (USoMS) corpus, a dataset of German PNs. We believe this resource could be used for experiments in the automatic extraction of emotion carriers from PN, a task that could provide further advancements in narrative understanding.

show abstract

“…For the manual annotation, we opted to use Appen crowdsourcing platform 6 . One of the challenges with crowdsourced annotation is to find a large number of qualified workers while filtering out low-quality workers or spammers (Chowdhury et al, 2015;Chowdhury et al, 2014) the quality of the annotation we followed standard evaluation (Chowdhury et al, 2020b), i.e., we used 150 gold standard test tweets. Based on these gold standard test tweets, each annotator needed to pass at least 70% of the tweets to participate in the annotation task.…”

Section: Manual Annotationmentioning

confidence: 99%

ArCovidVac: Analyzing Arabic Tweets About COVID-19 Vaccination

Mubarak¹,

Hassan²,

Chowdhury³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The emergence of the COVID-19 pandemic and the first global infodemic have changed our lives in many different ways. We relied on social media to get the latest information about COVID-19 pandemic and at the same time to disseminate information. The content in social media consisted not only health related advise, plans, and informative news from policymakers, but also contains conspiracies and rumors. It became important to identify such information as soon as they are posted to make an actionable decision (e.g., debunking rumors, or taking certain measures for traveling). To address this challenge, we develop and publicly release the first largest manually annotated Arabic tweet dataset, ArCovidVac, for the COVID-19 vaccination campaign, covering many countries in the Arab region. The dataset is enriched with different layers of annotation, including, (i) Informativeness (more vs. less importance of the tweets); (ii) fine-grained tweet content types (e.g., advice, rumors, restriction, authenticate news/information); and (iii) stance towards vaccination (pro-vaccination, neutral, anti-vaccination). Further, we performed in-depth analysis of the data, exploring the popularity of different vaccines, trending hashtags, topics and presence of offensiveness in the tweets. We studied the data for individual types of tweets and temporal changes in stance towards vaccine. We benchmarked the ArCovidVac dataset using transformer architectures for informativeness, content types, and stance detection.

show abstract

Cross-language transfer of semantic annotation via targeted crowdsourcing

Cited by 8 publications

References 12 publications

HumAID: Human-Annotated Disaster Incidents Data from Twitter with Deep Learning Benchmarks

HumAID: Human-Annotated Disaster Incidents Data from Twitter with Deep Learning Benchmarks

Annotation of Emotion Carriers in Personal Narratives

ArCovidVac: Analyzing Arabic Tweets About COVID-19 Vaccination

Contact Info

Product

Resources

About