Did the Models Understand Documents? Benchmarking Models for Language Understanding in Document-Level Relation Extraction

Abstract

Document-level relation extraction (DocRE) attracts more research interest recently. While models achieve consistent performance gains in DocRE, their underlying decision rules are still understudied: Do they make the right predictions according to rationales? In this paper, we take the first step toward answering this question and then introduce a new perspective on comprehensively evaluating a model. Specifically, we first conduct annotations to provide the rationales considered by humans in DocRE. Then, we conduct investigations and reveal the fact that: In contrast to humans, the representative state-of-the-art (SOTA) models in DocRE exhibit different decision rules. Through our proposed RE-specific attacks, we next demonstrate that the significant discrepancy in decision rules between models and humans severely damages the robustness of models and renders them inapplicable to real-world RE scenarios. After that, we introduce mean average precision (MAP) to evaluate the understanding and reasoning capabilities of models. According to the extensive experimental results, we finally appeal to future work to consider evaluating both performance and the understanding ability of models for the development of their applications. We make our annotations and code publicly available 1 .

Rethinking the Development of Large Language Models from the Causal Perspective: A Legal Text Prediction Case Study

“…However, in the meantime, some severe problems of models are also discovered, including unfairness and discrimination (Chalkidis et al 2022b). Accordingly, researchers propose debiasing methods (Guo, Yang, and Abbasi 2022;Sevim, S ¸ahinuc ¸, and Koc ¸2022) to mitigate the bias or conduct substantial experiments to investigate and analyze the decision rules of PLMs (Clark et al 2019;Chen, Chen, and Zhou 2023). Different from previous work, we rethink the development of LLMs from the causal perspective to theoretically analyze the underlying causes of their problems.…”

“…Large language models (LLMs) have undergone significant development and significantly impacted our life in a wide range of applications (OpenAI 2023), including legal judgment prediction (Feng, Li, and Ng 2022;Chalkidis et al 2022a), drug discovery (Singha Roy and Mercer 2023), and quantitative trading (Sawhney et al 2021;Ju et al 2023). While we enjoy their human-surpassing performance, they exhibit certain risks caused by confusing causality from correlation (Chen, Chen, and Zhou 2023). Their failure of learning causality (rationales) not only degrades their performance but also renders them untrustworthy, thus impeding their real-world applications, especially in those high- stake tasks that require rationales for decisions (e.g., legal judgment prediction).…”

Three-stage document-level entity relation extraction

“…To sum up, most researches in document relation inference treat document relation extraction as a separate problem: through the content of the article to infer The entity pair relationship Chen et al (2023). That is to say, the model requires both a certain understanding ability and a certain reasoning ability, which is relatively high performance requirements for a single model.…”