Discovering Black Lives Matter Events in the United States: Shared Task 3, CASE 2021

Abstract

Evaluating the state-of-the-art event detection systems on determining spatio-temporal distribution of the events on the ground is performed unfrequently. But, the ability to both (1) extract events "in the wild" from text and (2) properly evaluate event detection systems has potential to support a wide variety of tasks such as monitoring the activity of sociopolitical movements, examining media coverage and public support of these movements, and informing policy decisions. Therefore, we study performance of the best event detection systems on detecting Black Lives Matter (BLM) events from tweets and news articles. The murder of George Floyd, an unarmed Black man, at the hands of police officers received global attention throughout the second half of 2020. Protests against police violence emerged worldwide and the BLM movement, which was once mostly regulated to the United States, was now seeing activity globally. This shared task asks participants to identify BLM related events from large unstructured data sources, using systems pretrained to extract socio-political events from text. We evaluate several metrics, assessing each system's ability to evolution of protest events both temporally and spatially. Results show that identifying daily protest counts is an easier task than classifying spatial and temporal protest trends simultaneously, with maximum performance of 0.745 (Spearman) and 0.210 (Pearson r), respectively. Additionally, all baselines and participant systems suffered from low recall (max.5.08), confirming the high impact of media sourcing in the modelling of protest movements.

Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021): Workshop and Shared Task Report

“…The participants of Task 1 are invited to run the systems they develop to tackle Task 1 on a news and a Twitter archive. This is a similar setting with the edition performed last year in the scope of CASE 2021 and reported by Giorgi et al (2021). This year's results 4 are reported by Zavarella et al (2022).…”

“…We hypothesize that there is a low coverage of the Telegram dataset this year, since in the 2021 issue of this task (Giorgi et al, 2021), participating systems achieved much higher correlation with the ACLED Gold standard, including the NEXUS system. The low correlation can be explained with the low coverage of the war, where battles are not always reported in the media, especially in Telegram.…”

DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain

“…With the introduction of BERT (Devlin et al, 2019), transformer models have achieved state-of-the-art results in different NLP applications such as text classification (Ranasinghe and Hettiarachchi, 2020), information extraction (Plum et al, 2022) and event detection (Giorgi et al, 2021). Furthermore, the transformer architectures have shown promising results in SQuAD dataset (Zhang et al, 2021;Zhang et al, 2020;Yamada et al, 2020;Lan et al, 2020).…”