Textometr: an online tool for automated complexity level assessment of texts for Russian language learners

Abstract

Evaluation of text accessibility seems to be an extremely urgent and labor-consuming task in the process of preparing texts for teaching Russian as a foreign language. On the other hand, the procedure of assigning a text to one of the levels on the CEFR scale (from A1 to C2) is well-formalized and described in the professional literature, which opens opportunities for its automation. This paper presents Textometr - a new free web-based tool for estimating CEFR level and other key statistics from any given text in Russian that can be relevant for adapting it for foreign students. The automated assessment of the text level here is based on a regression model, trained on the dataset of more than 800 texts from Russian textbooks for foreigners, applying several machine learning and natural language processing methods. In addition to the CEFR level, the tool provides information relevant for adapting the text to educational tasks: lists of keywords and words for a potential vocabulary list, statistics on the text coverage by frequency lists and CEFR-graded vocabulary lists (lexical minima), a frequency list of the text, a forecast of the time needed for reading. The tool shortages at the current stage of development and suggested ways to solve them are also discussed. Finally, the results of the test on the tool quality and the vectors for its further development are reported. Textometr can provide helpful information not only to teachers and guidance teachers, but to authors of textbooks and publishers to check the compliance of the text content with the declared level and educational goals.

Text complexity as a non-discrete value: Russian L2 text complexity dataset annotation based on Elo rating system

“…One of the main purposes of this dataset was to be a test set for the algorithm of the text complexity assessment for Russian L2 learners. In the previous study we developed the ML system trained on 800 texts from Russian L2 textbooks and a set of linguistic features, including lexical, morphological, grammatical, and syntactic ones (Laposhina et al 2018). The examples of linguistic features are shown in Table 2.…”

“…An automated approach to the complexity assessment of the Russian L2 texts has several examples, most of them are based on datasets with discrete levels, such as the corpus of textbooks annotated by publishers on the CEFR scale can be used (Reynolds 2016;Karpov et al 2014;Batinic et al 2016;Laposhina et al 2018;Corlatescu et al 2022). However, to create a non-discrete scale, expert annotation is necessary, which can be time-consuming and expensive.…”

How do Eye Movements and Visual Attention in Letter-Finding Tasks Differ in Children with Different Levels of Reading Skills?

“…Text with a readability level 39 % (for grades 3-4) [14] and a list of letters displayed on the screen were used as stimulus material. The children read the text twice: in the first reading, the participants were not tasked with finding the letter; in the second reading, the participants were tasked with finding all the letters D. After the first reading of the text, the participants were told what task they would be asked to do on the second reading of the text.…”

Linguo-Pedagogical Design of Open Question Items in the Context of Digitalization

“…Среди сервисов анализа русскоязычных текстов можно выделить «Текстометр» и «Простой русский». Первый основан на модели машинного обучения и предполагает анализ лексических, грамматических и синтаксических признаков текста [22]; второй реализован в рамках концепции «простого языка» (plain language) и основан на статистистических расчётах по формулам читабельности, адаптированным для русскоязычных текстов [23]. В качестве примера работы сервисов приведём результаты анализа следующего вопроса к тестовому заданию: Как называется твёрдая фракция, содержащая органоминеральные вещества, выделяемые биоценозом активного ила в процессе его жизнедеятельности при реализации технологии биологической обработки сточных вод?…”