Neural Language Modeling for Contextualized Temporal Graph Generation

Abstract

This paper presents the first study on using large-scale pre-trained language models for automated generation of an event-level temporal graph for a document. Despite the huge success of neural pre-training methods in NLP tasks, its potential for temporal reasoning over event graphs has not been sufficiently explored. Part of the reason is the difficulty in obtaining large training corpora with humanannotated events and temporal links. We address this challenge by using existing IE/NLP tools to automatically generate a large quantity (89,000) of system-produced document-graph pairs, and propose a novel formulation of the contextualized graph generation problem as a sequence-to-sequence mapping task. These strategies enable us to leverage and fine-tune pre-trained language models on the systeminduced training data for the graph generation task. Our experiments show that our approach is highly effective in generating structurally and semantically valid graphs. Further, evaluation on a challenging hand-labeled, out-ofdomain corpus shows that our method outperforms the closest existing method by a large margin on several metrics. We also show a downstream application of our approach by adapting it to answer open-ended temporal questions in a reading comprehension setting. 1

Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning

“…Representative works on graph generation from language models include knowledge graph completion models like Comet Hwang et al, 2021) that fine-tune GPT (Radford et al, 2019;Brown et al, 2020) and BART (Lewis et al, 2020), generation of event influence graphs (Tandon et al, 2019;Madaan et al, 2020), partially ordered scripts (Sakaguchi et al, 2021), temporal graphs (Madaan and Yang, 2021), entailment trees , proof graphs (Saha et al, 2020;Saha et al, 2021a) and commonsense explanation graphs (Saha et al, 2021b). Linguistic tasks like syntactic parsing Mohammadshahi and Henderson, 2021;Kondratyuk and Straka, 2019) and semantic parsing (Chen et al, 2020b;Shin et al, 2021) have also made use of language models.…”

“…We test the generalizability of constructing structurally and semantically perturbed graphs for contrastive learning by also experimenting on a temporal graph generation task (Madaan and Yang, 2021) that requires constructing a temporal graph from a document. The nodes in the graph are events from the document and the edges are temporal relations between events ("before", "after", etc).…”

“…Train Dev Test ExplaGraphs 2368 398 400 Temporal (Sampled) 9531 953 949 Table 7: Train, validation and test split sizes of the two datasets. For Temporal Graph Generation, we randomly sample 1.3% of the overall corpus (Madaan and Yang, 2021).…”

“…Following our overall goal of improving graph generation in limited data settings, we randomly sample 1.3% of the overall corpus (∼ 9.5k samples) as the training corpus such that all graphs are connected DAGs. 5 Following Madaan and Yang (2021), we represent graphs in DOT format (Koutsofios and North, 1996) as shown in Fig. 4.…”

Could you give me a hint ? Generating inference graphs for defeasible reasoning

“…steps describing how viruses spread) and U i and H i are nodes of G i (these are phrases as shown in Figure 2). The output seq i op is set to a DOT-string representation of the corresponding influence graph G i , as such a representation was shown to be effective at extracting high-quality graphs (Madaan and Yang, 2021) from free-form text using language models (examples in the appendix). Thus, each passage-graph pair (T i , G i ) from WIQA is mapped to an input-output pair D = (seq i ip , seq i op ).…”