HaSpeeDe 2 @ EVALITA2020: Overview of the EVALITA 2020 Hate Speech Detection Task

Sanguinetti, Manuela; Comandini, Gloria; Nuovo, Elisa Di; Frenda, Simona; Stranisci, Marco; Bosco, Cristina; Caselli, Tommaso; Patti, Viviana; Russo, Irene

doi:10.4000/books.aaccademia.6897

Cited by 40 publications

(19 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the held-out test sets for Italian (Sanguinetti et al, 2020) and Portuguese (Fortuna et al, 2019), Perspective scored 70.7 and 64.1 macro F1. Perspective is outperformed on both languages by XTC, which scored 76.3 and 84.7 (Table 6).…”

Section: H Google Perspective Resultsmentioning

confidence: 99%

“…We fine-tune XLM-T on three widely-used hate speech datasets -one Spanish , one Italian (Sanguinetti et al, 2020) and one Portuguese (Fortuna et al, 2019). Accordingly, model performance is many-shot for Spanish, Italian and Portuguese, and zero-shot for all other languages.…”

Section: Multilingual Transformer Modelsmentioning

confidence: 99%

“…The Spanish dataset contains 4,950 tweets, of which 41.5% are labelled as hateful. The Italian Sanguinetti et al (2020) dataset contains 8,100 tweets, of which 41.8% are labelled as hateful. The Portuguese Fortuna et al (2019) dataset contains 5,670 tweets, of which 31.5% are labelled as hateful.…”

Section: Multilingual Transformer Modelsmentioning

confidence: 99%

“…The "Contro l'Odio" tweets were annotated by crowdworkers, but inter-annotator agreement was not reported. (Sanguinetti et al, 2020).…”

Section: Annotator Disagreement On Mhcmentioning

confidence: 99%

“…We denote the three XLM-T models trained on Italian Sanguinetti et al (2020), Portuguese Fortuna et al (2019 and Spanish as XLM-IT, XLM-PT and XLM-ES respectively. XTC denotes the XLM-T model trained on the combination of all three datasets, for which we report results in the main body of this article.…”

Section: F Xlm-t Model Comparisonmentioning

confidence: 99%

See 4 more Smart Citations

HateCheck: Functional Tests for Hate Speech Detection Models

Röttger¹,

Vidgen²,

Nguyen³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

102

View full text Add to dashboard Cite

Detecting online hate is a difficult task that even state-of-the-art models struggle with. Typically, hate speech detection models are evaluated by measuring their performance on held-out test data using metrics such as accuracy and F1 score. However, this approach makes it difficult to identify specific model weak points. It also risks overestimating generalisable model performance due to increasingly well-evidenced systematic gaps and biases in hate speech datasets. To enable more targeted diagnostic insights, we introduce HATECHECK, a suite of functional tests for hate speech detection models. We specify 29 model functionalities motivated by a review of previous research and a series of interviews with civil society stakeholders. We craft test cases for each functionality and validate their quality through a structured annotation process. To illustrate HATECHECK's utility, we test near-state-of-the-art transformer models as well as two popular commercial models, revealing critical model weaknesses.

show abstract

Section: H Google Perspective Resultsmentioning

confidence: 99%

Section: Multilingual Transformer Modelsmentioning

confidence: 99%

Section: Multilingual Transformer Modelsmentioning

confidence: 99%

“…The "Contro l'Odio" tweets were annotated by crowdworkers, but inter-annotator agreement was not reported. (Sanguinetti et al, 2020).…”

Section: Annotator Disagreement On Mhcmentioning

confidence: 99%

Section: F Xlm-t Model Comparisonmentioning

confidence: 99%

See 3 more Smart Citations

HateCheck: Functional Tests for Hate Speech Detection Models

Röttger¹,

Vidgen²,

Nguyen³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

102

View full text Add to dashboard Cite

show abstract

When Sarcasm Hurts: Irony-Aware Models for Abusive Language Detection

Frenda,

Patti,

Rosso

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Automatische Klassifikation offensiver deutscher Sprache in sozialen Netzwerken

Demus¹,

Labudde²,

Pitz³

et al. 2023

Digitale Hate Speech

View full text Add to dashboard Cite

ZusammenfassungDer Umgang mit Hatespeech ist bereits seit mehreren Jahren ein Problem im Internet, insbesondere in sozialen Netzwerken. Da die enorme Menge an Kommentaren nicht mehr manuell moderiert werden kann, ist es essenziell, automatische Methoden zur Detektion offensiver Kommentare unterstützend einzusetzen. Doch speziell in Bezug auf die deutsche Sprache bringt die Erforschung von Methoden zur Hatespeech-Erkennung einige Schwierigkeiten mit sich: zum einen sprachliche Besonderheiten und zum anderen die Knappheit geeigneter Datensätze. Deshalb soll mit diesem Kapitel ein Überblick über die Forschungsentwicklung gegeben werden, die wir insbesondere anhand von Shared Tasks darstellen. Außerdem werden geeignete Datensätze, Methoden und Ergebnisse zusammenfassend dargestellt und diskutiert.

show abstract

HaSpeeDe 2 @ EVALITA2020: Overview of the EVALITA 2020 Hate Speech Detection Task

Cited by 40 publications

References 31 publications

HateCheck: Functional Tests for Hate Speech Detection Models

HateCheck: Functional Tests for Hate Speech Detection Models

When Sarcasm Hurts: Irony-Aware Models for Abusive Language Detection

Automatische Klassifikation offensiver deutscher Sprache in sozialen Netzwerken

Contact Info

Product

Resources

About