Language Resources for Japanese Lexical Simplification

Abstract

This study introduces three language resources for Japanese lexical simplification: 1) an evaluation dataset, 2) lexica, and 3) a toolkit that can be used to develop and benchmark Japanese lexical simplification systems. The word complexity lexicon adopted in this study was automatically expanded using a classifier trained on a small word complexity lexicon created by Japanese language teachers. Based on this word complexity estimator, simplified word pairs were extracted from a large-scale synonym lexicon, and a simplified synonym lexicon that is useful for lexical simplification was developed. In addition, a Python library, which implements automatic evaluation and key methods in each subtask to ease the construction process of a lexical simplification pipeline, was developed. The experimental results on the developed evaluation dataset revealed that the proposed method, which is based on the developed lexicon, achieves the highest performance of Japanese lexical simplification.

Lexical Complexity Prediction: An Overview

“…We presented various use cases and applications of lexical complexity prediction, including for other NLP-related tasks such as sentiment analysis (Section 5.3.1), author identification (Section 5.3.2), and machine translation (Section 5.3.3). We collected and summarized English datasets used for LCP (Section 6.1) and also briefly presented work on languages other than English (Section 6.2): Chinese [76], Japanese [93], and Swedish [124].…”

“…JEV contains 18,000 thousand Japanese words divided into three levels of difficulty: easy, medium, or difficult. Nishihara and Kajiwara [93] also rated the complexity of words from Japanese Wikiepedia, the Tsukuba Web Corpus [94], and the Corpus of Contemporary Written Japanese [83]. This increased the size of their dataset to 40,605 Japanese words.…”

“…9.2.3 Japanese. Nishihara and Kajiwara [111] used a SVM to predict the lexical complexities of Japanese words. They created a new dataset that expanded upon the Japanese Education Vocabulary List (JEV).…”

“…However, they discarded other popular features, such as word length, due to the topological and morphological differences between English and Japanese. Unlike English, Japanese "is composed of three types of characters: Hiragana, Katakana, and Kanji" [111]. The characters Hiragana and Katakana are considered simple characters, whereas Kanji are ideographic and are therefore considered more difficult to interpret.…”

“…As such, in Japanese, the proportion of complex to simple characters within a word is a good indicator of a word's complexity. Nishihara and Kajiwara [111] concluded that the use of such language specific features was responsible for their model's good performance. 9.2.4 Swedish.…”