Heterogeneous Graph Neural Networks to Predict What Happen Next

Abstract

Given an incomplete event chain, script learning aims to predict the missing event, which can support a series of NLP applications. Existing work cannot well represent the heterogeneous relations and capture the discontinuous event segments that are common in the event chain. To address these issues, we introduce a heterogeneous-event (HeterEvent) graph network. In particular, we employ each unique word and individual event as nodes in the graph, and explore three kinds of edges based on realistic relations (e.g., the relations of word-and-word, word-and-event, event-and-event). We also design a message passing process to realize information interactions among homo or heterogeneous nodes. And the discontinuous event segments could be explicitly modeled by finding the specific path between corresponding nodes in the graph. The experimental results on one-step and multi-step inference tasks demonstrate that our ensemble model HeterEvent [W +E] can outperform existing baselines.

Incorporating Circumstances into Narrative Event Prediction

“…The UniFA-S (Zheng et al, 2020a) adopted Variational AutoEncoder architecture(Kingma and Welling, 2014) with a unified fine-tuning method to learn event representation from intra-inter-event and scenario level. HeterEvent (Zheng et al, 2020b) proposed a heterogeneous graph neural network that models discontinuous event segments explicitly. In the external knowledge enriched representation research line, researchers have attempted several external knowledge into event representation.…”

“…Besides, Lv et al (2019) employs a self-attention mecha-nism (Lin et al, 2017) to represent the event chain in diverse event segments within the chain implicitly. Zheng et al (2020b) adopt the graph attention network (Velickovic et al, 2018) to aggregate neighborhood events information. We employ the multi-head attention (Vaswani et al, 2017) to extract circumstance representation from the event sentence and aggregate circumstances in the global level adaptively.…”

“…Each argument has n args words. We follow (Zheng et al, 2020b) to apply a max-pooling and an average pooling layer on argument word embeddings and then concatenate them to get the event argument embeddings. The subject, predicate and the object representaion are denoted as s(e), p(e), o(e) ∈ R 2de .…”

“…• EventTransE (Lee and Goldwasser, 2019) is an representation learning method that explores discourse relations among events. • HeterEvent [W +E] (Zheng et al, 2020b) is a representation learning method, which adopts heterogeneous event graph to capture the discontinuous event segments explicitly. • UniFA-S (Zheng et al, 2020a) is a representation learning method based on the variational auto-encoder, which fine-tunes the pre-trained BERT (Devlin et al, 2019) on the NYT corpus and the event chains in multi-steps.…”

“…Lee and Goldwasser (2019) regards event embedding learning as a multi-relational problem and captures different relations of events pairs, such as the cause and contrast. Zheng et al (2020b) builds a heterogeneous event graph to mining subordinated relations between events and words.…”