Semantic Structural Evaluation for Text Simplification

Abstract

Current measures for evaluating text simplification systems focus on evaluating lexical text aspects, neglecting its structural aspects. In this paper we propose the first measure to address structural aspects of text simplification, called SAMSA. It leverages recent advances in semantic parsing to assess simplification quality by decomposing the input based on its semantic structure and comparing it to the output. SAMSA provides a reference-less automatic evaluation procedure, avoiding the problems that reference-based methods face due to the vast space of valid simplifications for a given sentence. Our human evaluation experiments show both SAMSA's substantial correlation with human judgments, as well as the deficiency of existing reference-based measures in evaluating structural simplification. 1

Content Differences in Syntactic and Semantic Representation

“…While graduation evokes a Scene (Figure 1), in UD it is an oblique modifier of moved, just like Paris is (Figure 2). The Scene/non-Scene distinction ( §5.2) would assist structural text simplification systems in paraphrasing this sentence to two sentences, each one containing one Scene (Sulem et al, 2018a).…”

“…It aims to represent the main semantic phenomena in text, abstracting away from syntactic forms. Shown to be preserved remarkably well across translations (Sulem et al, 2015), it has been applied to improve text simplification (Sulem 1 , 2018b), and text-to-text generation evaluation (Birch et al, 2016;Choshen and Abend, 2018;Sulem et al, 2018a). Formally, UCCA structures are directed acyclic graphs (DAGs) whose nodes (or units) correspond either to words, or to elements viewed as a single entity according to some semantic or cognitive consideration.…”

SemEval-2019 Task 1: Cross-lingual Semantic Parsing with UCCA

“…Required by some structural pattern. tion (Sulem et al, 2018b), as well as for defining semantic evaluation measures for text-to-text generation tasks, including machine translation (Birch et al, 2016), text simplification (Sulem et al, 2018a) and grammatical error correction (Choshen and Abend, 2018). The shared task defines a number of tracks, based on the different corpora and the availability of external resources (see §5).…”

Made for Each Other: Broad-Coverage Semantic Structures Meet Preposition Supersenses

“…UCCA's foundational layer, which is the only layer annotated over text so far, 3 reflects a coarse-grained level of semantics that has been shown to be preserved remarkably well across translations (Sulem et al, 2015). It has also been successfully used for improving text simplification (Sulem et al, 2018b), as well as to the evaluation of a number of textto-text generation tasks (Birch et al, 2016;Sulem et al, 2018a;Choshen and Abend, 2018).…”