Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence

Abstract

Automated event extraction in social science applications often requires corpus-level evaluations: for example, aggregating text predictions across metadata and unbiased estimates of recall. We combine corpus-level evaluation requirements with a real-world, social science setting and introduce the INDIAPO-LICEEVENTS corpus-all 21,391 sentences from 1,257 English-language Times of India articles about events in the state of Gujarat during March 2002. Our trained annotators read and label every document for mentions of police activity events, allowing for unbiased recall evaluations. In contrast to other datasets with structured event representations, we gather annotations by posing natural questions, and evaluate off-the-shelf models for three different tasks: sentence classification, document ranking, and temporal aggregation of target events. We present baseline results from zero-shot BERT-based models fine-tuned on natural language inference and passage retrieval tasks. Our novel corpus-level evaluations and annotation approach can guide creation of similar social-science-oriented resources in the future.

Zero-Shot Ranking Socio-Political Texts with Transformer Language Models to Reduce Close Reading Time

“…We compare the results by checking how much recall is achieved when a specified proportion of data is read from the ranked documents following Halterman et. al.…”

“…We carried out the experiments on two different datasets: India Police Events dataset 1 (Halterman et al, 2021) and the ProtestNews dataset of the workshop CASE @ ACL-IJCNLP 2021 2 (Hür- 1. A document belongs to a class if any of its sentences belongs to that class.…”

“…as the declarative query for the ProtestNews dataset. For the India Police Events dataset, we use declarative queries proposed by Halterman et al (2021).…”

“…Baeza- Yates and Ribeiro-Neto (1999) propose ranking documents in decreasing order of being relevant to a given query to accelerate the information retrieval process. Halterman et al (2021) apply this method with Natural Language Understanding (NLU) models for binary classification problems using the entailment probabilities of a document and a declarative form of the label. Therefore, to catch a high percentage of positively labeled documents, reading some percentage of documents would be enough since documents that are relevant would be at the top with a high probability.…”

“…We apply this approach to ProtestNews dataset along with the India Police Events dataset (Halterman et al, 2021) and investigate whether sentence level evaluation or document level evaluation ranks positive documents at the higher level measured by different evaluation metrics. We further investigate whether using the dictionary definition of a class or the declarative form of a class for the query performs this task better.…”