Real-time web text classification and analysis of reading difficulty

Abstract

The automatic analysis and categorization of web text has witnessed a booming interest due to the increased text availability of different formats, content, genre and authorship. We present a new tool that searches the web and performs in real-time a) html-free text extraction, b) classification for thematic content and c) evaluation of expected reading difficulty. This tool will be useful to adolescent and adult low-level reading students who face, among other challenges, a troubling lack of reading material for their age, interests and reading level.

Reconstructing Readability: Recent Developments and Recommendations in the Analysis of Text Difficulty

“…Future empirical research will determine the predictive validity of Read-X and its usability in educational settings. Meanwhile, developers are planning to examine whether psycholinguistic and discourse processing factors like syntactic complexity, propositional density, and rhetorical structure may improve the program's readability analysis (Miltsakaki and Troutt 2008).…”

Assessing user-specific difficulty of documents

“…A downside of the indirect approach, as can also be seen in the results of this study, is that the accuracy is not as good as in semi-manual systems. The method presented in this article can be used also, for example, in competence management (Honkela, Nordfors, & Tuuli, 2004;Volpentesta & Felicetti, 2011) or in a search engine for educational material for adolescents or younger pupils (Miltsakaki & Troutt, 2008). A broad scenario is to use the method to assess documents retrieved by a general-purpose search engine.…”

“…One approach to automate readability calculations is to select those formulas that do not require linguistic resources (e.g., Miltsakaki & Troutt, 2008;Petersen & Ostendorf, 2009;Crossley, Allen, & McNamara, 2011). Another possibility is to use extensive language-specific preprocessing or word lists (Feng, Elhadad, & Huenerfauth, 2009;François, 2009;Heilman, Collins-Thompson, & Eskenazi, 2008).…”

Web page Classification by Using PCA and Neural Network

“…Every classifier operates well on different aspects of the training or test feature vector. As a result, assuming appropriate conditions, combining multiple classifiers may improve classification performance when compared with any single classifier (Miltsakaki andTroutt, 2009 andWikipedia, 2013).…”