Automated Thematic Coherence Scoring of Student Essays Written in Portuguese

Abstract

While Thematic Coherence is a fundamental aspect of essay writing, scoring it is labor-intensive. This issue is often addressed using machine learning algorithms to estimate the score. However, related work is mostly limited to the English language or argumentative essays. Consequently, there is a lack of research on other widely used languages and essay types, such as Brazilian Portuguese and narrative essays. Hence, this paper reports on the findings of a study that aimed to evaluate the value of machine learning algorithms to automatically score the Thematic Coherence of both narratives (n = 400) and argumentative (n = 6567) essays written in Brazilian Portuguese. Expanding on previous studies, this paper evaluated regression models using conventional, feature-based algorithms according to essays’ linguistic features. Overall, we found that Extra Trees was the best performing algorithm, yielding predictions with moderate to strong correlations with human-generated scores. Mainly, those findings expand the literature with evidence on the potential of machine learning to estimate the Thematic Coherence of narrative and argumentative essays, suggest an improved performance for the former type.