Constructing and validating readability models: the method of integrating multilevel linguistic features with machine learning

Sung, Yao–Ting; Chen, Ju Ling; Her, Ji; Tseng, Hou Chiang; Chang, Tao; Chang, Kuo En

doi:10.3758/s13428-014-0459-x

Cited by 37 publications

(22 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also noticed that among the 30 selected features only the top 25 were included to create a readability model that is valid for categorizing CFL texts to the accuracy rate of 75% (exact accuracy) and 99% (adjacent accuracy), compared to a baseline of 44% and 87%, respectively. The superior performance of our model over the baseline lends support to those researchers who claimed that a small number of features may not represent the complex cognitive process of reading comprehension (Bailin & Grafstein, ; Bruce et al, ; Graesser et al., ; Sung et al., ) because we included a higher number of features as well as more complex features than the baseline.…”

Section: Discussionsupporting

confidence: 73%

Leveling L2 Texts Through Readability: Combining Multilevel Linguistic Features with the CEFR

Sung

Lin

Dyson

et al. 2015

The Modern Language Journal

Self Cite

View full text Add to dashboard Cite

Selecting appropriate texts for L2 (second/foreign language) learners is an important approach to enhancing motivation and, by extension, learning. There is currently no tool for classifying foreign language texts according to a language proficiency framework, which makes it difficult for students and educators to determine the precise difficulty/complexity levels of an unclassified text. Taking the Chinese language as an example, this study aimed to create a readability assessment system, called the Chinese Readability Index Explorer for Chinese as a Foreign Language (CRIE-CFL), in order to level-that is, to sort by proficiency level-texts that will be used for instructional purposes. The framework of choice in this project is the Common European Framework of Reference (CEFR). A team of expert CFL teachers first classified 1,578 CFL texts into their appropriate CEFR levels. A set of 30 CFL readability features was then developed or drawn from previous research, and sorted according to importance using F-scores. In addition, a support vector machine model was trained by sequentially integrating the features into the model to optimize accuracy. The empirical evaluation of CRIE-CFL revealed average exact-and adjacent-level accuracies of 74.97% and 99.62%, respectively, for predicting the expert classification of a text. The functionalities of CRIE-CFL are introduced and discussed.

show abstract

Section: Discussionsupporting

confidence: 73%

Leveling L2 Texts Through Readability: Combining Multilevel Linguistic Features with the CEFR

Sung

Lin

Dyson

et al. 2015

The Modern Language Journal

Self Cite

View full text Add to dashboard Cite

show abstract

“…Using the SVM model along with 24 linguistic features, the readability prediction of language arts textbooks used in grades 1-6 reached an accuracy of 72.92 %. Sung, Chen, et al (2015) confirmed that the prediction accuracy of the SVM model along with 32 linguistic features outperformed the prediction accuracy of discriminate analysis models in classifying language arts textbooks used in grades 1-6. In a preliminary study, Sung, Lin, and Tseng (2014) further found that CRIE 2.0 achieved similar results for language arts texts used in grades 1-9.…”

Section: Crie 20supporting

confidence: 54%

“…Previous research Sung, Chen, et al, 2015) validated the effectiveness of CRIE 2.0. Using the SVM model along with 24 linguistic features, the readability prediction of language arts textbooks used in grades 1-6 reached an accuracy of 72.92 %.…”

Section: Crie 20mentioning

confidence: 91%

CRIE: An automated analyzer for Chinese texts

Sung

Chang²,

Lin

et al. 2015

Behav Res

Self Cite

View full text Add to dashboard Cite

Textual analysis has been applied to various fields, such as discourse analysis, corpus studies, text leveling, and automated essay evaluation. Several tools have been developed for analyzing texts written in alphabetic languages such as English and Spanish. However, currently there is no tool available for analyzing Chinese-language texts. This article introduces a tool for the automated analysis of simplified and traditional Chinese texts, called the Chinese Readability Index Explorer (CRIE). Composed of four subsystems and incorporating 82 multilevel linguistic features, CRIE is able to conduct the major tasks of segmentation, syntactic parsing, and feature extraction. Furthermore, the integration of linguistic features with machine learning models enables CRIE to provide leveling and diagnostic information for texts in language arts, texts for learning Chinese as a foreign language, and texts with domain knowledge. The usage and validation of the functions provided by CRIE are also introduced.

show abstract

“…As Sebastiani (2001) argued, the algorithm generated by such a machine-learning approach often achieves accuracy comparable to that of human experts. Automatic text classification by machine learning is used in various fields of text analysis, such as scoring student essays (Hastings, Hughes, Magliano, Goldman, & Lawless, 2012), testing readability of texts (Sung et al, 2015), and investigating grammatical development of child language (Hassanali, Liu, Iglesias, Solorio, & Dollaghan, 2014). The attempts to develop a classifier of both suicidal notes and depressed tweets have been successful as well, achieving approximately 80% prediction accuracy (De Choudhury et al 2013;Pestian et al 2010).…”

mentioning

confidence: 99%

Unraveling the linguistic nature of specific autobiographical memories using a computerized classification algorithm

et al. 2016

View full text Add to dashboard Cite

In the present study, we explored the linguistic nature of specific memories generated with the Autobiographical Memory Test (AMT) by developing a computerized classifier that distinguishes between specific and nonspecific memories. The AMT is regarded as one of the most important assessment tools to study memory dysfunctions (e.g., difficulty recalling the specific details of memories) in psychopathology. In Study 1, we utilized the Japanese corpus data of 12,400 cuerecalled memories tagged with observer-rated specificity. We extracted linguistic features of particular relevance to memory specificity, such as past tense, negation, and adverbial words and phrases pertaining to time and location. On the basis of these features, a support vector machine (SVM) was trained to classify the memories into specific and nonspecific categories, which achieved an area under the curve (AUC) of .92 in a performance test. In Study 2, the trained SVM was tested in terms of its robustness in classifying novel memories (n = 8,478) that were retrieved in response to cue words that were different from those used in Study 1. The SVM showed an AUC of .89 in classifying the new memories. In Study 3, we extended the binary SVM to a five-class classification of the AMT, which achieved 64%-65% classification accuracy, against the chance level (20%) in the performance tests. Our data suggest that memory specificity can be identified with a relatively small number of words, capturing the universal linguistic features of memory specificity across memories in diverse contents. Keywords Autobiographical memory . Natural language processing . Machine learning . Support vector machineThe nature of specific autobiographical memoriesThe specificity of autobiographical memories has been studied frequently in the past three decades, in the fields of cognitive, social, and clinical psychology. An autobiographical memory (AM) refers to Ba memory that is concerned with the recollection of personally experienced past events ( Williams et al., 2007, p. 122). The level of episodic specificity with which such personal memories are recalled has become a target of increased interest related to topics such as aging, cultural differences, and-most importantly-psychopathology (e.g., Addis, Wong, & Schacter, 2008;van Vreeswijk & de Wilde, 2004;Wang, Hou, Tang, & Wiprovnick, 2011). Specificity is important not only for vividly reexperiencing past events, but also for clearly imagining future events. In fact, remembering past events and envisioning possible future events share similar cognitive functions and neural substrates (Addis, Wong, & Schacter, 2007;D'Argembeau & Van der Linden, 2006;Okuda et al. 2003). Thus, AM recall, and particularly the retrieval of episodic specificity, is assumed to play a central role in human functioning because AM specificity contributes to a sense of self and serves as a source for future planning and goal pursuit .Reduced AM specificity is considered particularly important in clinical psychology. Empirical studies in ...

show abstract

Constructing and validating readability models: the method of integrating multilevel linguistic features with machine learning

Cited by 37 publications

References 42 publications

Leveling L2 Texts Through Readability: Combining Multilevel Linguistic Features with the CEFR

Leveling L2 Texts Through Readability: Combining Multilevel Linguistic Features with the CEFR

CRIE: An automated analyzer for Chinese texts

Unraveling the linguistic nature of specific autobiographical memories using a computerized classification algorithm

Contact Info

Product

Resources

About