Enhancing Pointer Network for Sentence Ordering with Pairwise Ordering Predictions

Abstract

Dominant sentence ordering models use a pointer network decoder to generate ordering sequences in a left-to-right fashion. However, such a decoder only exploits the noisy left-side encoded context, which is insufficient to ensure correct sentence ordering. To address this deficiency, we propose to enhance the pointer network decoder by using two pairwise ordering prediction modules: The FUTURE module predicts the relative orientations of other unordered sentences with respect to the candidate sentence, and the HISTORY module measures the local coherence between several (e.g., 2) previously ordered sentences and the candidate sentence, without the influence of noisy left-side context. Using the pointer mechanism, we then incorporate this dynamically generated information into the decoder as a supplement to the left-side context for better predictions. On several commonly-used datasets, our model significantly outperforms other baselines, achieving the state-of-the-art performance. Further analyses verify that pairwise ordering predictions indeed provide extra useful context as expected, leading to better sentence ordering. We also evaluate our sentence ordering models on a downstream task, multi-document summarization, and the summaries reordered by our model achieve the best coherence scores. Our code is available at https://github.com/DeepLearnXMU/Pairwise.git.

An External Knowledge Enhanced Graph-based Neural Network for Sentence Ordering

“…In the future, we plan to design a fault-tolerant architecture to mitigating error propagation from external tools. Besides, incorporating more powerful decoding algorithms (Zhang et al, 2018a;Yin et al, 2020a) is a promising direction.…”

BERT-enhanced Relational Sentence Ordering Network

“…We compute such pairwise relational representation for all the sentence pairs in the paragraph, and utilize the subset of them at each step of the decoder. Different from the previous method of using the learned sentence vectors to calculate the pairwise relationship between sentences (Yin et al, 2020), DRM employs the whole sequence of the sentence pair as the input to BERT. It allows us to directly relate words from different sentences together, which is more straightforward to exploit the intrinsic relations and coherence between sentences.…”

“…It allows us to directly relate words from different sentences together, which is more straightforward to exploit the intrinsic relations and coherence between sentences. Further, instead of relying on the modules trained from scratch to control the pairwise ordering predictions (Yin et al, 2020), DRM adopts BERT as the main building block to obtain a pairwise relationship representation for the sentence pair. Intuitively, being pre-trained on the large corpus in BERT, this representation encodes more reliable and accurate relative ordering information, and thus is more effective to help determine the pairwise ordering predictions in the decoder.…”

“…(2) Pointer network based models: HAN (Wang and Wan, 2019); LSTM+PtrNet (Gong et al, 2016); V-LSTM+PtrNet (Logeswaran et al, 2018); ATTOrderNet (Cui et al, 2018); SE-Graph (Yin et al, 2019); FUDecoder (Yin et al, 2020); TGCM (Oh et al, 2019).…”

“…It is able to capture the paragraphlevel contextual information for generating an ordered sequence with the highest coherence probability (Gong et al, 2016;Logeswaran et al, 2018;Cui et al, 2018;Yin et al, 2019). Further, HAN (Wang and Wan, 2019) and TGCM (Oh et al, 2019) introduce the attention mechanism (Vaswani et al, 2017), and FUDecoder (Yin et al, 2020) proposes pairwise ordering prediction modules to enhance the traditional pointer network.…”