SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)

Abstract

We present the results and the main findings of SemEval-2019 Task 6 on Identifying and Categorizing Offensive Language in Social Media (OffensEval). The task was based on a new dataset, the Offensive Language Identification Dataset (OLID), which contains over 14,000 English tweets. It featured three sub-tasks. In sub-task A, the goal was to discriminate between offensive and non-offensive posts. In sub-task B, the focus was on the type of offensive content in the post. Finally, in sub-task C, systems had to detect the target of the offensive posts. OffensEval attracted a large number of participants and it was one of the most popular tasks in SemEval-2019. In total, about 800 teams signed up to participate in the task, and 115 of them submitted results, which we present and analyze in this report.

Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack

“…Further, data collected in the context of a dialogue rather than a sentence without context provides more sophisticated attacks. We show that model architectures that use the dialogue context efficiently perform much bet-ter than systems that do not, where the latter has been the main focus of existing research (Wulczyn et al, 2017;Davidson et al, 2017;Zampieri et al, 2019).…”

“…The task of detecting offensive language has been studied across a variety of content classes. Perhaps the most commonly studied class is hate speech, but work has also covered bullying, aggression, and toxic comments (Zampieri et al, 2019).…”

Neural Word Decomposition Models for Abusive Language Detection

“…Identifying abusive context on the web is one of the widely studied topics on social media text. This problem has been studied for Hate Speech detection (Kwok and Wang, 2013;Waseem and Hovy, 2016;Waseem, 2016;Ross et al, 2016;Saleem et al, 2017;Warner and Hirschberg, 2012), Harassment (Yin et al, 2009;Cheng et al, 2015), Cyberbullying (Willard, 2007;Tokunaga, 2010;Schrock and Boyd, 2011), Abusive language detection (Sahlgren et al, 2018;Nobata et al, 2016), aggression identification (Kumar et al, 2018;Aroyehun and Gelbukh, 2018;Modha et al, 2018), identifying toxic comments on forums (Wulczyn et al, 2017) and offensive language identification (Wiegand et al, 2018;Zampieri et al, 2019). While most of the work in identifying abusive on social media is predominantly studied for English social media posts some of the latest work include study on German (Wiegand et al, 2018), Italian (Bosco et al, 2018) and Mexican Spanish (Álvarez-Carmona et al, 2018).…”

“…While Badjatiya et al (2017) analyzed various architectures to encode text for hatespeech detection, we are not aware of any work that studied various word decomposition models for identifying abusive language in text. Recent work on identifying offensive language in text include fine-tuning large pretrained languege model BERT which use subword units to encode text (Zampieri et al, 2019;Zhu et al, 2019). For the SEMEVAL-2019 task of offensive language identification 7 out of top 10 submissions used BERT finet tuning.…”

“…For the SEMEVAL-2019 task of offensive language identification 7 out of top 10 submissions used BERT finet tuning. Zampieri et al (2019) highlighted that 8% of 104 systems participated in the shared task used BERT based fine-tuning.…”