Error Analysis in an Automated Narrative Information Extraction Pipeline

Valls-Vargas, Josep; Zhu, Jiasong; Ontañón, Santiago

doi:10.1109/tciaig.2016.2575823

Cited by 8 publications

(7 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Aside from the original MUC-4 evaluation scoring reports (Chinchor, 1991), which included counts of missing and spurious role filler errors, there have been very few attempts at understanding the types of errors made by IE systems and grounding those errors linguistically. Valls-Vargas et al (2017) proposed a framework for studying how different errors propagate through an IE system; however, the framework can only be used for pipelined systems, not end-to-end ones.…”

Section: Related Workmentioning

confidence: 99%

Automatic Error Analysis for Document-level Information Extraction

Das¹,

Du²,

Wang³

et al. 2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Document-level information extraction (IE)tasks have recently begun to be revisited in earnest using the end-to-end neural network techniques that have been successful on their sentence-level IE counterparts. Evaluation of the approaches, however, has been limited in a number of dimensions. In particular, the precision/recall/F1 scores typically reported provide few insights on the range of errors the models make. We build on the work of Kummerfeld and Klein (2013) to propose a transformation-based framework for automating error analysis in document-level event and (N-ary) relation extraction. We employ our framework to compare two state-of-theart document-level template-filling approaches on datasets from three domains; and then, to gauge progress in IE since its inception 30 years ago, vs. four systems from the MUC-4 (1992) evaluation. 1 * These authors contributed equally to this work. 1 Our code for the error analysis tool and its output on different model predictions are available at https://github. com/IceJinx33/auto-err-template-fill/.

show abstract

Section: Related Workmentioning

confidence: 99%

Automatic Error Analysis for Document-level Information Extraction

Das¹,

Du²,

Wang³

et al. 2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

show abstract

“…Her application Creative Help is a story assisting tool similar to LISA, yet it differs in functionality by predicting stories rather than checking the existing story for inconsistencies. Josep Valls-Vargas et al introduce a system Vox which performs information and character role extraction (Valls-Vargas, Zhu, and Ontanon 2016). This work is highly useful for extracting information from the story regarding characters and events, and would extend LISA to be used along with a character-centric system.…”

Section: Related Workmentioning

confidence: 99%

LISA: Lexically Intelligent Story Assistant

Sanghrajka

Hidalgo

Chen

et al. 2021

AIIDE

View full text Add to dashboard Cite

This paper serves as an introduction to building an assistive tool for story writers. Our tool, Lexically Intelligent Story Assistant (or LISA), aims to assist story writers by providing real-time feedback on lexical inconsistencies in the story. LISA analyzes the narrative and builds a knowledge base, using artificial intelligence to make inferences and point out errors in the narrative. Moreover, it also allows the user to interact with the system by querying the knowledge base in natural language form. This tool shows that it is possible to create a database for a narrative and use artificial intelligence to improve authoring of stories.

show abstract

“…Inspired by Valls-Vargas et al [27], we consider the notion of error propagation: given a sub-pipeline (c i , . .…”

Section: Measuring Quality In a Pipelinementioning

confidence: 99%

Medical Dialogue Summarization for Automated Reporting in Healthcare

Molenaar

Maas

Burriel

et al. 2020

Lecture Notes in Business Information Processing

View full text Add to dashboard Cite

Healthcare providers generally spend excessive time on administrative tasks at the expense of direct patient care. The emergence of new artificial intelligence and natural language processing technologies gives rise to innovations that could relieve them of this burden. In this paper, we present a pipeline structure for building dialogue summarization systems. Our pipeline summarizes a consultation of a patient with a care provider and automatically generates a report compliant with medical formats. Four pipeline components are used to generate a report based on audio input. The outputs of each component are analyzed to determine the most important challenges and issues. The current proof-of-concept, which was applied to eight doctor-to-patient sessions concerning ear infection, shows that automatic dialogue summarization and reporting is achievable, but requires improvements to increase completeness.

show abstract

Error Analysis in an Automated Narrative Information Extraction Pipeline

Cited by 8 publications

References 22 publications

Automatic Error Analysis for Document-level Information Extraction

Automatic Error Analysis for Document-level Information Extraction

LISA: Lexically Intelligent Story Assistant

Medical Dialogue Summarization for Automated Reporting in Healthcare

Contact Info

Product

Resources

About