Explaining Errors in Predictions of At-Risk Students in Distance Learning Education

Exploring critical factors of the perceived usefulness of a learning analytics dashboard for distance university students

“…It is used across all faculties in the undergraduate modules of the OU and is currently only available to teachers. A detailed description of the LAD and the algorithm behind it can be found in our published work (Huptych et al, 2017 ; Hlosta et al, 2020 ). This study was the first attempt to share the LAD with the students and enable them to see and comment on their own LA data.…”

Making sense of learning data at scale

“…Rather than showing to stakeholders what is all possible (as the OU collects a lot of data on learners and learning), we focused our initial development of the dashboards based upon what was first and foremost relevant to teachers. Initially, these dashboards just provided key metrics and drill-downs of main data that teachers considered to be relevant, while over time these led to more advanced learning analytics systems like OU Analyse, forecasting future learner performance (Hlosta et al, 2020), as explained in detail in Box 1.…”

“…One of the smart tricks of OU Analyse is that its combination of machine learning approaches can predict which learning activities are key for successful completion of an assessment, and which ones are not so relevant, as well as how a learner has previously performed. Based upon thousands of successful and unsuccessful learner paths, OU Analyse gives information to the teacher whether or not key learning activities have been undertaken by Agnes Hlosta et al, 2020). While early predictions until week 16 indicated uncertainty about whether Agnes would submit, the submission of TMA2 and accessing the homepage activity in week 14 eventually led OU Analyse to assume that she would submit TMA3, which indeed she did with a good assignment score of 80%.…”

Teachers’ Perspectives on the Promises, Needs and Challenges of Learning Analytics Dashboards: Insights from Institutions Offering Blended and Distance Learning

“…No model can predict outcomes with absolute certainty, and there will always be things that affect students' learning and performance that are beyond the university's control or knowledge, such as a change in personal circumstances (Hlosta et al, 2020). However, the predictive models used combine the effects of multiple factors to create their probabilities and have been shown to provide an acceptable level of accuracy at the individual student level (Hlosta et al, 2017).…”

“…In recognising that sometimes predictions are erroneous, Hlosta et al (2020) carried out a mixed methods study with first-year OU STEM students to identify students who were incorrectly identified as at risk of not submitting their next assignment but did submit (FN), and those who were not identified as at risk of non-submission but did not submit (FP), based on the findings from OUA shortterm predictions. Between 2017 and 2019, they identified 38,073 predictions over 17 modules that met the criteria and concentrated on predictions 2 weeks prior to the submission date of the student's first assignment.…”