Deep learning for early performance prediction of introductory programming students: a comparative and explanatory study

Abstract

Introductory programming may be complex for many students. Moreover, there is a high failure and dropout rate in these courses. A potential way to tackle this problem is to predict student performance at an early stage, as it facilitates human-AI collaboration towards prescriptive analytics, where the instructors/monitors will be told how to intervene and support students - where early intervention is crucial. However, the literature states that there is no reliable predictor yet for programming students’ performance, since even large-scale analysis of multiple features have resulted in only limited predictive power. Notice that Deep Learning (DL) can provide high-quality results for huge amount of data and complex problems. In this sense, we employed DL for early prediction of students’ performance using data collected in the very first two weeks from introductory programming courses offered for a total of 2058 students during 6 semesters (longitudinal study). We compared our results with the state-of-the-art, an Evolutionary Algorithm (EA) that automatic creates and optimises machine learning pipelines. Our DL model achieved an average accuracy of 82.5%, which is statistically superior to the model constructed and optimised by the EA (p-value << 0.05 even with Bonferroni correction). In addition, we also adapted the DL model in a stacking ensemble for continuous prediction purposes. As a result, our regression model explained ~62% of the final grade variance. In closing, we also provide results on the interpretation of our regression model to understand the leading factors of success and failure in introductory programming.

Explaining Individual and Collective Programming Students’ Behavior by Interpreting a Black-Box Predictive Model

“…In previous works [45], [47], [48], we have composed a set of data-driven features that, in conjunction, have a high predictive power to infer the students' performance, even when using early data, from the first two weeks of a course. Our previous ML model [48] achieved an average accuracy of 78.2% with this early data, outperforming cutting edge results for this task [1], [12], [18], [20], [30], [33], [53], [68], [71].…”

“…Their results pointed to tree-based ensembles being more suitable for our data. As an extension, in [45] we surpassed [48], obtaining an average accuracy of 82.2%, by using a deep learning architecture. Between the model presented in this current paper using XGBoost (with average accuracy of 81.3%) and our previous best result using deep learning, we did not find statistical significance (p-value>0.05).…”

“…However, the literature still lacked a reliable method to predict CS1 students' performance [58]. We thus proposed a potential solution to this gap [45], [47], [48] by composing a set of data-driven features (collected from literature and extended with self-devised ones), which we showed to have a high predictive power to infer the students' performance early on (from data from the first two weeks of the CS1 course).…”

Mapeamento e análise empírica de misconceptions comuns em avaliações de introdução à programação

“…Embora nomes diferentes sejam usados para representar as dificuldades dos alunos em aprender a programar, há um consenso geral de que é importante ajudar os alunos a desenvolver sua compreensão conceitual e superar desafios [4,14,15,23,27,28,30,35]. Para isso, é vital para o ensino das disciplinas introdutórias de programação saber quais são os misconceptions comuns [1,[14][15][16] para dar suporte aos professores, já que com essa informação os docentes podem adaptar sua metodologia com aulas que possam potencialmente minimizar a ocorrência desses erros de compreensão, atuando assim de forma preventiva.…”

“…Para isso, é vital para o ensino das disciplinas introdutórias de programação saber quais são os misconceptions comuns [1,[14][15][16] para dar suporte aos professores, já que com essa informação os docentes podem adaptar sua metodologia com aulas que possam potencialmente minimizar a ocorrência desses erros de compreensão, atuando assim de forma preventiva. Além disso, uma análise dos erros comuns pode propiciar uma intervenção precoce manual ou automática, aplicando-se abordagens e ferramentas instrucionais eficazes para dirimir potenciais dificuldades enfrentadas pelos alunos [22,23,35,36,40]. Diante dos potenciais benefícios, em um estudo recente, Qian et al [30] apontam que existe uma escassez de instrumentos de medição concretos e estudos empíricos que mapeiem e avaliem misconceptions comuns.…”