Automatic Error Analysis for Document-level Information Extraction

Das, Arundhoti; Du, Xuebin; Wang, Barry; Shi, Kun; Gu, Jiwei; Porter, Thomas R.; Cardie, Claire

doi:10.18653/v1/2022.acl-long.274

Cited by 6 publications

(4 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, it has become standard to evaluate only on the string-fill slots, plus the template type (Chambers and Jurafsky, 2011;Du et al, 2021b;Das et al, 2022;Chen et al, 2023). Du et al (2021b) thus proposed a version of Eq.…”

Section: Muc-4: Recent Workmentioning

confidence: 99%

A Unified View of Evaluation Metrics for Structured Prediction

Chen,

Gantt,

Chen

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

We present a conceptual framework that unifies a variety of evaluation metrics for different structured prediction tasks (e.g. event and relation extraction, syntactic and semantic parsing). Our framework requires representing the outputs of these tasks as objects of certain data types, and derives metrics through matching of common substructures, possibly followed by normalization. We demonstrate how commonly used metrics for a number of tasks can be succinctly expressed by this framework, and show that new metrics can be naturally derived in a bottom-up way based on an output structure. We release a library that enables this derivation to create new metrics. 1 Finally, we consider how specific characteristics of tasks motivate metric design decisions, and suggest possible modifications to existing metrics in line with those motivations. * Equal contribution.1 https://github.com/wanmok/metametric.2 Often in the form of a directed acyclic graph (DAG), as in the task of AMR parsing.

show abstract

Section: Muc-4: Recent Workmentioning

confidence: 99%

A Unified View of Evaluation Metrics for Structured Prediction

Chen,

Gantt,

Chen

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…In NLP, there are some task-specific works on automatic error analysis such as Das et al (2022) on document-level information extraction, Kummerfeld and Klein (2013) on coreference resolution, Popović and Ney (2011) on machine translation, and etc. There is also extensive research conducted on different model evaluations to see whether models make erroneous datapoints in certain types of noising datapoints (Belinkov and Bisk, 2017;Rychalska et al, 2019) or adversarial datapoints (Ribeiro et al, 2018;Iyyer et al, 2018).…”

Section: Related Workmentioning

confidence: 99%

Discover, Explanation, Improvement: Automatic Slice Detection Framework for Natural Language Processing

Hua¹,

Jin²,

Song³

et al. 2022

Preprint

View full text Add to dashboard Cite

Current natural language processing (NLP) models such as BERT and RoBERTa have achieved high overall performance, but they often make systematic errors due to bias or certain difficult features to learn. Thus research on slice detection models (SDM) which automatically identifies underperforming groups of datapoints has gradually caught more attention, which aims at both understanding model behaviors and providing insights for future model training and designing. However, there is little systematic research on SDM and quantitative evaluation of its assessment for NLP models. Our paper fills this gap by proposing "Discover, Explanation, Improvement (DEI)" framework that discovers coherent and underperforming groups of datapoints and unites datapoints of each slice under human-understandable concepts; it also provides comprehensive evaluation tasks and the corresponding quantitative metrics, which enable convenient comparison for future works. Results show that our framework can accurately select error-prone datapoints with informative semantic features that summarize error patterns, based on which it directly improves model performance by an average of 2.85 points without tuning any parameters.

show abstract

“…Nevertheless, many IE tasks require a more comprehensive understanding that often extends to the entire input document, leading to challenges such as length and multiple events when embedding full documents. Consequently, document-level datasets continue to pose challenges for even the most advanced models today (Das et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

Probing Representations for Document-level Event Extraction

Wang,

Du,

Cardie

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

The probing classifiers framework has been employed for interpreting deep neural network models for a variety of natural language processing (NLP) applications. Studies, however, have largely focused on sentencelevel NLP tasks. This work is the first to apply the probing paradigm to representations learned for document-level information extraction (IE). We designed eight embedding probes to analyze surface, semantic, and event-understanding capabilities relevant to document-level event extraction. We apply them to the representations acquired by learning models from three different LLM-based document-level IE approaches on a standard dataset. We found that trained encoders from these models yield embeddings that can modestly improve argument detections and labeling but only slightly enhance event-level tasks, albeit trade-offs in information helpful for coherence and event-type prediction. We further found that encoder models struggle with document length and cross-sentence discourse.

show abstract

Automatic Error Analysis for Document-level Information Extraction

Cited by 6 publications

References 23 publications

A Unified View of Evaluation Metrics for Structured Prediction

A Unified View of Evaluation Metrics for Structured Prediction

Discover, Explanation, Improvement: Automatic Slice Detection Framework for Natural Language Processing

Probing Representations for Document-level Event Extraction

Contact Info

Product

Resources

About