Multilingual Protest News Detection - Shared Task 1, CASE 2021

Abstract

Benchmarking state-of-the-art text classification and information extraction systems in multilingual, cross-lingual, few-shot, and zeroshot settings for socio-political event information collection is achieved in the scope of the shared task Socio-political and Crisis Events Detection at the workshop CASE @ ACL-IJCNLP 2021. Socio-political event data is utilized for national and international policyand decision-making. Therefore, the reliability and validity of such datasets are of utmost importance. We split the shared task into three parts to address the three aspects of data collection (Task 1), fine-grained semantic classification (Task 2), and evaluation (Task 3). Task 1, which is the focus of this report, is on multilingual protest news detection and comprises four subtasks that are document classification (subtask 1), sentence classification (subtask 2), event sentence coreference identification (subtask 3), and event extraction (subtask 4). All subtasks have English, Portuguese, and Spanish for both training and evaluation data. Data in Hindi language is available only for the evaluation of subtask 1. The majority of the submissions, which are 238 in total, are created using multi-and cross-lingual approaches. Best scores are between 77.27 and 84.55 F1-macro for subtask 1, between 85.32 and 88.61 F1macro for subtask 2, between 84.23 and 93.03 CoNLL 2012 average score for subtask 3, and between 66.20 and 78.11 F1-macro for subtask 4 in all evaluation settings. The performance of the best system for subtask 4 is above 66.20 F1 for all available languages. Although there is still a significant room for improvement in cross-lingual and zero-shot settings, the best submissions for each evaluation scenario yield remarkable results. Monolingual models outperformed the multilingual models in a few evaluation scenarios, in which there is relatively much training data.

Utilizing coarse-grained data in low-data settings for event extraction

“…We utilize ACL CASE 2021 shared task [Hürriyetoglu et al, 2021] data, which is an event information detection shared task focusing on protest events. This data set consists of news articles from various countries published between 2000 and 2018 in various languages.…”

The Causal News Corpus: Annotating Causal Relations in Event Sentences from News

“…CNC builds on the datasets featured in a series of workshops directed at mining socio-political events from news articles: Automated Extraction of Socio-political Events from News (AESPEN) in 2020 (Hürriyetoglu et al, 2020b;Hürriyetoglu et al, 2020a) and Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) in 2021 (Hürriyetoglu et al, 2021a;Hürriyetoglu et al, 2021b;Hürriyetoglu, 2021). The data is based on randomly sampled articles from multiple sources and periods, and all annotations were performed by two annotators, adjudicated and spot-checked by a supervisor, and corrected further semi-automatically (Hürriyetoglu et al, 2021c;Yörük et al, 2021).…”

“…Despite the importance of identifying causality in text, datasets are limited (Asghar, 2016;Xu et al, 2020;Tan et al, 2021;Yang et al, 2021) and oftentimes, different researchers craft their datasets with different rules, leaving users with no proper way to compare models across datasets. Our work is directed at annotating parts of the multilingual protest news detection dataset (Hürriyetoglu et al, 2021a;Hürriyetoglu et al, 2021b;Hürriyetoglu, 2021) for event causality. Our contributions 1 are as follows:…”

Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021): Workshop and Shared Task Report

“…The extended multilingual protest news detection is the same shared task organized at CASE 2021 (Hürriyetoglu et al, 2021a). This year we introduced additional data and languages at the evaluation stage.…”