A Systematic Study of Neural Discourse Models for Implicit Discourse Relation

Abstract

Inferring implicit discourse relations in natural language text is the most difficult subtask in discourse parsing. Many neural network models have been proposed to tackle this problem. However, the comparison for this task is not unified, so we could hardly draw clear conclusions about the effectiveness of various architectures. Here, we propose neural network models that are based on feedforward and long-short term memory architecture and systematically study the effects of varying structures. To our surprise, the best-configured feedforward architecture outperforms LSTM-based model in most cases despite thorough tuning. Further, we compare our best feedforward system with competitive convolutional and recurrent networks and find that feedforward can actually be more effective. For the first time for this task, we compile and publish outputs from previous neural and nonneural systems to establish the standard for further comparison.

Mining Discourse Markers for Unsupervised Sentence Representation Learning

“…The PDTB is annotated with a hierarchy of relations, with 5 classes at level 1 (including the EntRel relation), and 16 at level 2 (with one relation absent from the test). It is interesting to see that this form of simple semi-supervised learning for implicit relation prediction performs quite well, especially for fine-grained relations, as the best model slightly beats the best current dedicated model, listed at 40.9% in Rutherford et al (2017).…”

Zero-shot transfer for implicit discourse relation classification

“…The PDTB framework allows annotations to be labelled with more than one label. In such cases we only keep the first label, in line with previous studies (among others Ji and Eisenstein, 2015;Rutherford et al, 2017).…”

“…The main purpose of this study is to assess the performance of transfer learning on the implicit discourse relation classification task. To this end, we use a simple feedforward network fed with multilingual sentence embeddings following the finding of (Rutherford et al, 2017) which shows that simple discourse models with feedforward layers perform on par or better than those of with surface features or recurrent and convolutional architectures. We follow the model of due to its simplicity and robust nature even in the multilingual setting with different argument and discourse relation representations.…”

Do We Need Cross Validation for Discourse Relation Classification?

“…The architecture of the model we use is illustrated in Figure 1. Regarding the initialization, regularization and learning algorithm, we follow all the settings in (Rutherford et al, 2017). We adopt cross-entropy as our cost function, adagrad as the optimization algorithm, initialized all the weights in the model with uniform random and set dropout layers after the embedding and output layer with a drop rate of 0.2 and 0.5 respectively.…”