Hitachi at SemEval-2020 Task 11: An Empirical Study of Pre-Trained Transformer Family for Propaganda Detection

Morio, Gaku; Morishita, Terufumi; Ozaki, Hiroaki; Miyoshi, Takushi

doi:10.18653/v1/2020.semeval-1.228

Cited by 15 publications

(8 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Span prediction identifies the start and end offsets of the span (Chhablani et al 2021). Sequence labelling classifies each member of a sequence, for example, identify whether each token is toxic (Chhablani et al 2021) or use BIO encoding (i.e., mark the token as (B) if it is at the beginning, (I) if it is inside or (O) if it is outside of the span) (Morio et al 2020).…”

Section: Span Extractionmentioning

confidence: 99%

“…Since SE tasks require highly nuanced semantic understanding, most solutions leveraged large language models pre-trained using transformers, including BERT (Devlin et al 2019) and other types of transformers (Morio et al 2020;Chhablani et al 2021). These models are pre-trained on billions of words of English text data and can be easily fine-tuned to adapt to new tasks.…”

Section: Span Extractionmentioning

confidence: 99%

“…They experimented with an ensemble of a model trained on the shared task dataset and one on the combined dataset of the original and silver data produced using self-learning. Though they achieved promising results, their performance was poorer compared to the solution of Morio et al (2020). Although they also used an ensemble of pre-trained transformers, their success relied on a complex heterogeneous multi-layer neural network.…”

Section: Span Extractionmentioning

confidence: 99%

See 2 more Smart Citations

Automated hate speech detection and span extraction in underground hacking and extremist forums

et al. 2022

View full text Add to dashboard Cite

Hate speech is any kind of communication that attacks a person or a group based on their characteristics, such as gender, religion and race. Due to the availability of online platforms where people can express their (hateful) opinions, the amount of hate speech is steadily increasing that often leads to offline hate crimes. This paper focuses on understanding and detecting hate speech in underground hacking and extremist forums where cybercriminals and extremists, respectively, communicate with each other, and some of them are associated with criminal activity. Moreover, due to the lengthy posts, it would be beneficial to identify the specific span of text containing hateful content in order to assist site moderators with the removal of hate speech. This paper describes a hate speech dataset composed of posts extracted from HackForums, an online hacking forum, and Stormfront and Incels.co, two extremist forums. We combined our dataset with a Twitter hate speech dataset to train a multi-platform classifier. Our evaluation shows that a classifier trained on multiple sources of data does not always improve the performance compared to a mono-platform classifier. Finally, this is the first work on extracting hate speech spans from longer texts. The paper fine-tunes BERT (Bidirectional Encoder Representations from Transformers) and adopts two approaches – span prediction and sequence labelling. Both approaches successfully extract hateful spans and achieve an F1-score of at least 69%.

show abstract

Section: Span Extractionmentioning

confidence: 99%

Section: Span Extractionmentioning

confidence: 99%

Section: Span Extractionmentioning

confidence: 99%

See 1 more Smart Citation

Automated hate speech detection and span extraction in underground hacking and extremist forums

et al. 2022

View full text Add to dashboard Cite

show abstract

“…As previously introduced, current systems address these tasks relying on word embedding models (e.g., BERT-embedding) and standard features (e.g., PoS, name-entity, n-grams), as representations to feed various RNN architectures (Morio et al, 2020;Chernyavskiy et al, 2020). Recently, the language model BERT (Devlin et al, 2019) has been widely utilized to optimize the performances of classification tasks, but there is still room for improvement, in particular when applied to propaganda detection (Da San Martino et al, 2020a.…”

Section: Related Workmentioning

confidence: 99%

“Don’t discuss”: Investigating Semantic and Argumentative Features for Supervised Propagandist Message Detection and Classification

Vorakitphan

Cabrio

Villata

2021

Proceedings of the Conference Recent Advances in Natural Language Processing - Deep Learning for Natural Language Processing Me

View full text Add to dashboard Cite

One of the mechanisms through which disinformation is spreading online, in particular through social media, is by employing propaganda techniques. These include specific rhetorical and psychological strategies, ranging from leveraging on emotions to exploiting logical fallacies. In this paper, our goal is to push forward research on propaganda detection based on text analysis, given the crucial role these methods may play to address this main societal issue. More precisely, we propose a supervised approach to classify textual snippets both as propaganda messages and according to the precise applied propaganda technique, as well as a detailed linguistic analysis of the features characterising propaganda information in text (e.g., semantic, sentiment and argumentation features). Extensive experiments conducted on two available propagandist resources (i.e., show that the proposed approach, leveraging different language models and the investigated linguistic features, achieves very promising results on propaganda classification, both at sentenceand at fragment-level.

show abstract

“…The proposed approaches for the sub-task can be broadly classified into Span Prediction or Token Classification. Most teams use multi-granular transformer-based systems for token classification/sequence tagging (Khosla et al, 2020;Morio et al, 2020;Patil et al, 2020). Inspired by Souza et al (2019), Jurkiewicz et al (2020) use RoBERTa-CRF based systems.…”

Section: Introductionmentioning

confidence: 99%

NLRG at SemEval-2021 Task 5: Toxic Spans Detection Leveraging BERT-based Token Classification and Span Prediction Techniques

Chhablani,

Sharma,

Pandey

et al. 2021

Preprint

View full text Add to dashboard Cite

Toxicity detection of text has been a popular NLP task in the recent years. In SemEval-2021 Task-5 Toxic Spans Detection, the focus is on detecting toxic spans within passages. Most state-of-the-art span detection approaches employ various techniques, each of which can be broadly classified into Token Classification or Span Prediction approaches. In our paper, we explore simple versions of both of these approaches and their performance on the task. Specifically, we use BERT-based models -BERT, RoBERTa, and SpanBERT for both approaches. We also combine these approaches and modify them to bring improvements for Toxic Spans prediction. To this end, we investigate results on four hybrid approaches -Multi-Span, Span+Token, LSTM-CRF, and a combination of predicted offsets using union/intersection. Additionally, we perform a thorough ablative analysis and analyze our observed results. Our best submissiona combination of SpanBERT Span Predictor and RoBERTa Token Classifier predictionsachieves an F 1 score of 0.6753 on the test set. Our best post-eval F 1 score is 0.6895 on intersection of predicted offsets from top-3 RoBERTa Token Classification checkpoints. These approaches improve the performance by 3% on average than those of the shared baseline models -RNNSL and SpaCy NER.

show abstract

Hitachi at SemEval-2020 Task 11: An Empirical Study of Pre-Trained Transformer Family for Propaganda Detection

Cited by 15 publications

References 21 publications

Automated hate speech detection and span extraction in underground hacking and extremist forums

Automated hate speech detection and span extraction in underground hacking and extremist forums

“Don’t discuss”: Investigating Semantic and Argumentative Features for Supervised Propagandist Message Detection and Classification

NLRG at SemEval-2021 Task 5: Toxic Spans Detection Leveraging BERT-based Token Classification and Span Prediction Techniques

Contact Info

Product

Resources

About