Lexical Simplification with Pretrained Encoders

Abstract

Lexical simplification (LS) aims to replace complex words in a given sentence with their simpler alternatives of equivalent meaning. Recently unsupervised lexical simplification approaches only rely on the complex word itself regardless of the given sentence to generate candidate substitutions, which will inevitably produce a large number of spurious candidates. We present a simple LS approach that makes use of the Bidirectional Encoder Representations from Transformers (BERT) which can consider both the given sentence and the complex word during generating candidate substitutions for the complex word. Specifically, we mask the complex word of the original sentence for feeding into the BERT to predict the masked token. The predicted results will be used as candidate substitutions. Despite being entirely unsupervised, experimental results show that our approach obtains obvious improvement compared with these baselines leveraging linguistic databases and parallel corpus, outperforming the state-of-the-art by more than 12 Accuracy points on three well-known benchmarks.

Multi-Word Lexical Simplification

“…Our approach to the MWLS problem, called the Plainifier, is an extension of the unsupervised method for single word lexical simplification by Qiang et al (2020). Following their work, we generate candidate replacements using BERT predictions for a given context and rank them according to language-model probability, simplicity and similarity of meaning to the original text.…”

“…In step (a), a gap created by removing the replaced token is filled with a single [MASK], for which the predictions are acquired from Terse-BERT. This method is used by Qiang et al (2020) to obtain one-word candidates, such as The cat sleeps on the mat, and their likelihoods. The multi-words setting in Plainifer requires step (b), in which the gap is filled with two [MASK] elements and the best (according to ranking described in section 4.3) K predictions for the first position are obtained.…”

“…Secondly, we design a method for generating such simplifications automatically. Our solution, called the Plainifier, is inspired by a recently proposed method for LS utilising language models (Qiang et al, 2020), which we extend so that multi-word simplifications can be obtained. In order to encourage more research on the problem, we make the dataset 1 , the language model 2 and the Plainifer code 3 openly available.…”

Lexical Simplification System to Improve Web Accessibility

“…They also tend to change the meaning of the sentence and have problems dealing with ambiguous words [30] [31] [32]. However, recently, unsupervised approaches have been improved in this regard by allowing more detailed context information to be obtained [33]. On the other hand, hybrid strategies employ methods from both the previous two approaches, such as [34], which uses a corpus-based approach and a combination of a free lexicon, decision trees, and context-based rules.…”

Language Resources for Japanese Lexical Simplification

“…1 (Petersen and Ostendorf 2007) ( 3 http://www.dianamccarthy.co.uk/task10index.html 4 https://www.cs.york.ac.uk/semeval-2012/task1/ 5 http://people.cs.kuleuven.be/˜jan.debelder/lseval.zip 6 https://simple.wikipedia.org/wiki/Wikipedia:Basic English combined wordlist 7 https://cs.pomona.edu/˜dkauchak/simplification/ 1 (Zhu et al 2010) PWKP / WikiSmall 108K Wikipedia (Coster and Kauchak 2011) 137K Wikipedia (Xu, Callison-Burch, and Napoles 2015) Newsela 96K (Hwang, Hajishirzi, Ostendorf, and Wu 2015) 392K Wikipedia (Kajiwara and Komachi 2016) 493K Wikipedia (Zhang and Lapata 2017) WikiLarge 286K Wikipedia (Maruyama and Yamamoto 2018) SNOW T15 50K (Katsuta and Yamamoto 2018) SNOW T23 35K (Paetzold and Specia 2017b;Qiang, Li, Zhu, Yuan, and Wu 2020) LSeval (3)…”