Early Alert Systems During a Pandemic: A Simulation Study on the Impact of Concept Drift

Xu, Yang; Wilson, Kevin E. S.

doi:10.1145/3448139.3448190

Cited by 9 publications

(2 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By analyzing this data, universities can gain valuable insights and make data-driven decisions and policies [37]. For example, by applying machine learning (ML) models, we could predict students' final course grades [9,27,40,42], classify students into groups based on those likely to pass or fail a course [18], predict students at risk of dropping out [8,11,14,41], or measure indicators of 21st century skills acquisition [24]. This has broadened the interest in both Learning Analytics (LA) and Educational Data Mining (EDM) as fields of research and practice.…”

Section: Introductionmentioning

confidence: 99%

Assessing the Fairness of Course Success Prediction Models in the Face of (Un)equal Demographic Group Distribution

Deho¹

2023

Preprint

View full text Add to dashboard Cite

In recent years, predictive models have been increasingly used by education practitioners and stakeholders to leverage actionable insights to support student success. Usually, model selection (i.e., the decision of which predictive model to use) is largely based on the predictive performance of the models. Nevertheless, it has become important to consider fairness as an integral part of the criteria for model selection. Might a model be unfair towards certain demographic groups? Might it systematically perform poorly for certain demographic groups? Indeed, prior studies affirm this. Which model out of the lot should we choose then? Additionally, prior studies suggest demographic group imbalance in the training dataset to be a source of such unfairness. If so, would the fairness of the predictive models improve if the demographic group distribution in the training dataset becomes balanced? This study seeks to answer these questions. Firstly, we analyze the fairness of 4 of the commonly used state-of-the-art models to predict course success for 3 IT courses in a large public Australian university. Specifically, we investigate if the models serve different demographic groups equally. Secondly, to address the identified unfairness—supposedly caused by the demographic group imbalance—we train the models on 3 types of \textit{balanced data} and investigate again if the unfairness was mitigated. We found that none of the predictive models was consistently fair in all 3 courses. This suggests that model selection decision should be carefully made by both the researchers and stakeholders as the per the requirement of the domain of application. Furthermore, we found that balancing demographic groups (and class labels) is not enough—albeit can be an initial step—to ensure fairness of predictive models in education. An implication of this is that sometimes, the source of unfairness may not be immediately apparent. Therefore, “blindly” attributing the unfairness to demographic group imbalance may cause the unfairness to persist even when the data becomes balanced. We hope that our findings can guide practitioners and relevant stakeholders in making well-informed decisions

show abstract

Section: Introductionmentioning

confidence: 99%

Assessing the Fairness of Course Success Prediction Models in the Face of (Un)equal Demographic Group Distribution

Deho¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent applications include, but are not limited to, IoT systems [6,7], smart grids [8,9], 5G networks [10] and stock market [11,12]. A recent study also has investigated the impact of concept drift on early alert systems during the SARS-CoV-2 pandemic [13]. These explored systems share the non-stationarity property since they are characterized by continuous changes as they develop.…”

Section: Introductionmentioning

confidence: 99%

From Concept Drift to Model Degradation: An Overview on Performance-Aware Drift Detectors

Bayram¹,

Ahmed²,

Kassler³

2022

Preprint

View full text Add to dashboard Cite

The dynamicity of real-world systems poses a significant challenge to deployed predictive machine learning (ML) models. Changes in the system on which the ML model has been trained may lead to performance degradation during the system's life cycle. Recent advances that study non-stationary environments have mainly focused on identifying and addressing such changes caused by a phenomenon called concept drift. Different terms have been used in the literature to refer to the same type of concept drift and the same term for various types. This lack of unified terminology is set out to create confusion on distinguishing between different concept drift variants. In this paper, we start by grouping concept drift types by their mathematical definitions and survey the different terms used in the literature to build a consolidated taxonomy of the field. We also review and classify performance-based concept drift detection methods proposed in the last decade. These methods utilize the predictive model's performance degradation to signal substantial changes in the systems. The classification is outlined in a hierarchical diagram to provide an orderly navigation between the methods. We present a comprehensive analysis of the main attributes and strategies for tracking and evaluating the model's performance in the

show abstract

The impact of disadvantage on higher education engagement during different delivery modes: a pre- versus peri-pandemic comparison of learning analytics data

Summers

Higson

Moores

2022

Assessment & Evaluation in Higher Education

View full text Add to dashboard Cite

Early Alert Systems During a Pandemic: A Simulation Study on the Impact of Concept Drift

Cited by 9 publications

References 8 publications

Assessing the Fairness of Course Success Prediction Models in the Face of (Un)equal Demographic Group Distribution

Assessing the Fairness of Course Success Prediction Models in the Face of (Un)equal Demographic Group Distribution

From Concept Drift to Model Degradation: An Overview on Performance-Aware Drift Detectors

The impact of disadvantage on higher education engagement during different delivery modes: a pre- versus peri-pandemic comparison of learning analytics data

Contact Info

Product

Resources

About