Integrating Transformer and Paraphrase Rules for Sentence Simplification

Abstract

Sentence simplification aims to reduce the complexity of a sentence while retaining its original meaning. Current models for sentence simplification adopted ideas from machine translation studies and implicitly learned simplification mapping rules from normalsimple sentence pairs. In this paper, we explore a novel model based on a multi-layer and multi-head attention architecture and we propose two innovative approaches to integrate the Simple PPDB (A Paraphrase Database for Simplification), an external paraphrase knowledge base for simplification that covers a wide range of real-world simplification rules. The experiments show that the integration provides two major benefits: (1) the integrated model outperforms multiple stateof-the-art baseline models for sentence simplification in the literature (2) through analysis of the rule utilization, the model seeks to select more accurate simplification rules. The code and models used in the paper are available at https://github.com/ Sanqiang/text_simplification.

EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing

“…For SMT-based models, Zhu et al (2010) adopt a tree-based SMT model for sentence simplification; Woodsend and Lapata (2011) propose a quasi-synchronous grammar and use integer linear programming to score the simplification rules; Wubben et al (2012) employ a phrase-based MT model to obtain candidates and re-rank them based on the dissimilarity to the complex sentence; Narayan and Gardent (2014) develop a hybrid model that performs sentence splitting and deletion first and then re-rank the outputs similar to Wubben et al (2012); Xu et al (2016) propose SBMT-SARI, a syntax-based machine translation framework that uses an external knowledge base to encourage simplification. On the other side, many NMT-based models have also been proposed for sentence simplification: Nisioi et al (2017) employ vanilla recurrent neural networks (RNNs) on text simplification; Zhang and Lapata (2017) propose to use reinforcement learning methods on RNNs to optimize a specific-designed reward based on simplicity, fluency and relevancy; incorporate memory-augmented neural networks for sentence simplification; Zhao et al (2018) integrate the transformer architecture and PPDB rules to guide the simplification learning; Sulem et al (2018b) combine neural MT models with sentence splitting modules for sentence simplification.…”

Controllable Text Simplification with Lexical Constraint Loss

“…Many previous studies (Specia, 2010;Wubben et al, 2012;Xu et al, 2016;Nisioi et al, 2017;Zhang and Lapata, 2017;Vu et al, 2018;Guo et al, 2018;Zhao et al, 2018) in text simplification have trained machine translators on a monolingual parallel corpus consisting of complex-simple sentence pairs without considering the level of each sentence. Therefore, these text simplification models are ignorant regarding the sentence level.…”

Complexity-Weighted Loss and Diverse Reranking for Sentence Simplification

“…This work has been shown to get state-of-the-art results in an automatic evaluation, training on the WikiLarge dataset introduced by Zhang and Lapata (2017). Zhao et al (2018), however, does not perform a human evaluation, and restricting evaluation to automatic metrics is generally insufficient for comparing simplification models. Our model, in comparison, is able to generate shorter and simpler sentences according to Flesch-Kincaid grade level (Kincaid et al, 1975) and human judgments, and provide a comprehensive analysis using human evaluation and a qualitative error analysis.…”

“…Following previous work (Zhang and Lapata, 2017;Zhao et al, 2018), we use SARI as our main automatic metric for evaluation (Xu et al, 2016). 11 Specifically, SARI calculates how often a generated sentence correctly keeps, inserts, and deletes n-grams from the complex sentence, using the reference simple standard as the gold-standard, where 1 ≤ n ≤ 4.…”

“…One of the main limitations in applying standard Seq2Seq models to simplification is that these models tend to copy directly from the original complex sentence too often, as this is the most common operation in simplification. Several recent efforts have attempted to alleviate this problem using reinforcement learning (Zhang and Lapata, 2017) and memory augmentation (Zhao et al, 2018), but these systems often still produce outputs that are longer than the reference sentences. To avoid this problem, we propose to extend the generic Seq2Seq framework at both training and inference time by encouraging the model to choose simpler content words, and by effectively choosing an output based on a large set of can-didate simplifications.…”