New Benchmark Corpus and Models for Fine-grained Event Classification: To BERT or not to BERT?

Piskorski, Jakub; Haneczok, Jacek; Jacquet, Guillaume

doi:10.18653/v1/2020.coling-main.584

Cited by 21 publications

(19 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Although most of the reported work in this area focuses on processing English texts, and in particular, news-like texts as presented in Piskorski et al (2020), some efforts on event classification for non-English language were reported too. For instance, Sahoo et al (2020) introduced a benchmark corpus for fine-grained classification of natural and man-made disasters (28 types) for Hindi, accompanied with evaluation of deep learning baseline models for this task.…”

Section: Prior Workmentioning

confidence: 99%

“…For the training purposes the participants were allowed to either exploit any freely available existing event-annotated textual corpora and/or to exploit the short text snippets reporting events which are part of the large event database created by ACLED and which can be obtained from ACLED data portal 4 for research and academic purposes. Furthermore, the participants were also recommended to exploit as an inspiration the techniques for text normalization and cleaning of ACLED data, and some baseline classification models trained using ACLED data described in Piskorski et al (2020).…”

Section: Training Datamentioning

confidence: 99%

See 1 more Smart Citation

Fine-grained Event Classification in News-like Text Snippets - Shared Task 2, CASE 2021

Haneczok¹,

Jacquet²,

Piskorski³

et al. 2021

Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events From Text (CAS

Self Cite

View full text Add to dashboard Cite

This paper describes the Shared Task on Fine-grained Event Classification in News-like Text Snippets. The Shared Task is divided into three subtasks: (a) classification of text snippets reporting socio-political events (25 classes) for which vast amount of training data exists, although exhibiting different structure and style vis-a-vis test data, (b) enhancement to a generalized zero-shot learning problem, where 3 additional event types were introduced in advance, but without any training data ('unseen' classes), and (c) further extension, which introduced 2 additional event types, announced shortly prior to the evaluation phase. The reported Shared Task focuses on classification of events in English texts and is organized as part of the Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021), co-located with the ACL-IJCNLP 2021 Conference. Four teams participated in the task. Best performing systems for the three aforementioned subtasks achieved 83.9%, 79.7% and 77.1% weighted F 1 scores respectively.

show abstract

Section: Prior Workmentioning

confidence: 99%

Section: Training Datamentioning

confidence: 99%

Fine-grained Event Classification in News-like Text Snippets - Shared Task 2, CASE 2021

Haneczok¹,

Jacquet²,

Piskorski³

et al. 2021

Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events From Text (CAS

Self Cite

View full text Add to dashboard Cite

show abstract

“…Piskorkski refers to the bert-base model fine-tuned by Piskorski et al (2020), which was the top performer on test fold 1 in their experiments. Test scores for the Piskorski model trained on 10% of the training data are estimated from the graphs in Piskorski et al (2020). metrics reported on test fold 1.…”

Section: Modelsmentioning

confidence: 99%

IBM MNLP IE at CASE 2021 Task 2: NLI Reranking for Zero-Shot Text Classification

Barker¹,

Awasthy²,

Ni³

et al. 2021

Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events From Text (CAS

View full text Add to dashboard Cite

Supervised models can achieve very high accuracy for fine-grained text classification. In practice, however, training data may be abundant for some types but scarce or even nonexistent for others. We propose a hybrid architecture that uses as much labeled data as available for fine-tuning classification models, while also allowing for types with little (fewshot) or no (zero-shot) labeled data. In particular, we pair a supervised text classification model with a Natural Language Inference (NLI) reranking model. The NLI reranker uses a textual representation of target types that allows it to score the strength with which a type is implied by a text, without requiring training data for the types. Experiments show that the NLI model is very sensitive to the choice of textual representation, but can be effective for classifying unseen types. It can also improve classification accuracy for the known types of an already highly accurate supervised model. 1

show abstract

“…What all of the systems have in common is that they need a representation of text that is understandable for a computer. Piskorski et al (2020) showed that modern transformer embeddings are the best choice by comparing them to classic word embeddings and achieving superior results with them. Based on these findings, we decided to make use of them in our work too.…”

Section: Related Workmentioning

confidence: 99%

FKIE_itf_2021 at CASE 2021 Task 1: Using Small Densely Fully Connected Neural Nets for Event Detection and Clustering

Becker¹,

Krumbiegel²

2021

Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events From Text (CAS

View full text Add to dashboard Cite

In this paper we present multiple approaches for event detection on document and sentence level, as well as a technique for event sentence co-reference resolution. The advantage of our co-reference resolution approach, which handles the task as a clustering problem, is that we use a single neural net to solve the task, which stands in contrast to other clustering algorithms that often are build on more complex models. This means that we can set our focus on the optimization of a single neural network instead of having to optimize numerous different parameters. We use small densely connected neural networks and pretrained multilingual transformer embeddings in all subtasks. We use either document or sentence embeddings, depending on the task, and refrain from using word embeddings, so that the implementation of complicated network structures and unfolding of RNNs, which can deal with input of different sizes, is not necessary. We achieved an average macro F1 of 0.65 in subtask 1 (i.e., document level classification), and a macro F1 of 0.70 in subtask 2 (i.e., sentence level classification). For the co-reference resolution subtask, we achieved an average CoNLL-2012 score across all languages of 0.83.

show abstract

New Benchmark Corpus and Models for Fine-grained Event Classification: To BERT or not to BERT?

Cited by 21 publications

References 17 publications

Fine-grained Event Classification in News-like Text Snippets - Shared Task 2, CASE 2021

Fine-grained Event Classification in News-like Text Snippets - Shared Task 2, CASE 2021

IBM MNLP IE at CASE 2021 Task 2: NLI Reranking for Zero-Shot Text Classification

FKIE_itf_2021 at CASE 2021 Task 1: Using Small Densely Fully Connected Neural Nets for Event Detection and Clustering

Contact Info

Product

Resources

About