Syntactically Robust Training on Partially-Observed Data for Open Information Extraction

Abstract

Open Information Extraction models have shown promising results with sufficient supervision. However, these models face a fundamental challenge that the syntactic distribution of training data is partially observable in comparison to the real world. In this paper, we propose a syntactically robust training framework that enables models to be trained on a syntactic-abundant distribution based on diverse paraphrase generation. To tackle the intrinsic problem of knowledge deformation of paraphrasing, two algorithms based on semantic similarity matching and syntactic tree walking are used to restore the expressionally transformed knowledge. The training framework can be generally applied to other syntactic partial observable domains. Based on the proposed framework, we build a new evaluation set called CaRB-AutoPara, a syntactically diverse dataset consistent with the real-world setting for validating the robustness of the models. Experiments including a thorough analysis show that the performance of the model degrades with the increase of the difference in syntactic distribution, while our framework gives a robust boundary. The source code is publicly available at https://github.com/qijimrc/RobustOIE.

Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction

“…Despite the widespread interest in these benchmarks and the related OpenIE approaches provides promising results. However, the traditional peer-topeer matching-based evaluation can not measure the robustness of those approaches, where the syntax and expression may be various with underlying meaning (Qi et al, 2023). This work significantly fills the gap between traditional metrics and missed robustness evaluation for OpenIE and calls for more efforts in this research area.…”

“…In order to analyze the syntactic divergence in the cliques, we need a metric to measure the syntactic correlation between two sentences. A fast and effective algorithm is the HWS distance proposed in (Qi et al, 2023), which calculates the syntactic tree distance between two sentences based on a hierarchically weighted matching strategy, where smaller weights imply a greater focus on the comparison of skeletons. The value domain of this is [0, 1], where 1 indicates the farthest distance.…”

“…The revised Hierarchically Weighted Syntactic Distance Algorithm (HWS distance) is shown in algorithm 1. We fix the over-counting problem for repeated consecutive spans while preserving the efficiency with the same time complexity in the original work (Qi et al, 2023).…”

“…Research including these efforts has been devoted to evaluating the pairwise matching correctness between model extractions and golden facts on a sentence. However, the conventional evaluation benchmarks do not measure the robustness of models in the realistic open-world scenario, where the syntactic and expressive forms may vary under the same knowledge meaning (Qi et al, 2023). As shown in Figure 1, while the three sentences s 1 , s 2 , s 3 contain the same structured knowledge (a 1 , p, a 2 , a 3 ), the state-of-the-art model OpenIE6 successfully extracts facts (in green color) on sentence s 1 , but fails to predict arguments (in red color) on the other sentences due to the syntactic and expressive drifts.…”