Reimagining the machine learning life cycle to improve educational outcomes of students

Abstract

Machine learning (ML) techniques are increasingly prevalent in education, from their use in predicting student dropout to assisting in university admissions and facilitating the rise of massive open online courses (MOOCs). Given the rapid growth of these novel uses, there is a pressing need to investigate how ML techniques support long-standing education principles and goals. In this work, we shed light on this complex landscape drawing on qualitative insights from interviews with education experts. These interviews comprise in-depth evaluations of ML for education (ML4Ed) papers published in preeminent applied ML conferences over the past decade. Our central research goal is to critically examine how the stated or implied education and societal objectives of these papers are aligned with the ML problems they tackle. That is, to what extent does the technical problem formulation, objectives, approach, and interpretation of results align with the education problem at hand? We find that a cross-disciplinary gap exists and is particularly salient in two parts of the ML life cycle: the formulation of an ML problem from education goals and the translation of predictions to interventions. We use these insights to propose an extended ML life cycle, which may also apply to the use of ML in other domains. Our work joins a growing number of meta-analytical studies across education and ML research as well as critical analyses of the societal impact of ML. Specifically, it fills a gap between the prevailing technical understanding of machine learning and the perspective of education researchers working with students and in policy.

LAK of Direction

“…Past assessments of the quality of LA research for improving learning has found that evidence of this efficacy is weak (Ferguson & Clow, 2017;Larrabee Sønderlund et al, 2019). Given this, our findings suggest that LA research lacks clear direction toward addressing questions about learning, preferring instead to examine analytical approaches (see also Liu et al, 2022). This focus on examining "things that we can do" instead of "problems that we can solve" has been a chronic obstacle in the nearby field of educational technology research for the past 40 years (Clark, 1983;Reeves & Lin, 2020).…”

On the Actionability of Outcome Prediction

“…Although risk prediction has become ubiquitous in education and other domains, its effectiveness for improving outcomes has been called into question. A recent empirical qualitative study by Liu et al (2023) on machine learning applications in education found a significant gap between predictions and beneficent interventions.…”

“…"You don't improve things by predicting them better." -Education researcher on the value of predicting academic risk (Liu et al 2023) The study of interventions has been fundamental in the social sciences, statistics, and theoretical computer science (Rosenbaum and Rubin 1983;Pearl 1995;Rubin 2005;Peters, Janzing, and Schölkopf 2017;Hofman et al 2021). The set of techniques and applications for causal inference and analysis is vast, mostly notably including program evaluation and randomized controlled trials (Stephenson and Imrie The Thirty-Eighth AAAI Conference on Artificial Intelligence 1998; Deaton and Cartwright 2018), observational studies (Rosenbaum, Rosenbaum, and Briskman 2010) using modern ML techniques (Athey and Imbens 2016), adaptive trial designs (Collins, Murphy, and Strecher 2007;Montoya et al 2022), individual treatment effect and counterfactual inference (Shalit, Johansson, and Sontag 2017;Lei and Candès 2021;Bynum, Loftus, and Stoyanovich 2023).…”

An Artificial Intelligence (AI) Framework to Predict Operational Excellence: UAE Case Study

“…Figure 9 shows that the AI life cycle is made up of seven stages, which are inception, design and development, verification and validation, deployment, operation and monitoring, re-evaluation, and retirement. These stages are mapped into the ML pipeline cycle in this paper's research framework [58].…”

“…Figure 9 shows that the AI life cycle is made up of seven stages, which are inception, design and development, verification and validation, deployment, operation and monitoring, re-evaluation, and retirement. These stages are mapped into the ML pipeline cycle in this paper's research framework [58]. ISO/IEC 23053 includes several important aspects that should be considered throughout the AI life cycle and ML pipeline, like risk management, governance, security, privacy, accountability, transparency, explainability, safety, resilience, robustness, and fairness [50].…”