Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education

Abstract

Colleges have increasingly turned to predictive analytics to target at-risk students for additional support. Most of the predictive analytic applications in higher education are proprietary, with private companies offering little transparency about their underlying models. We address this lack of transparency by systematically comparing two important dimensions: (1) different approaches to sample and variable construction and how these affect model accuracy and (2) how the selection of predictive modeling approaches, ranging from methods many institutional researchers would be familiar with to more complex machine learning methods, affects model performance and the stability of predicted scores. The relative ranking of students’ predicted probability of completing college varies substantially across modeling approaches. While we observe substantial gains in performance from models trained on a sample structured to represent the typical enrollment spells of students and with a robust set of predictors, we observe similar performance between the simplest and the most complex models.

Finishing the Last Lap: Experimental Evidence on Strategies to Increase Attainment for Students Near College Completion

“…While we find no impacts of message outreach on persistence and completion across the distribution of predicted baseline risk, the null effects found in this study do not necessarily reflect that predictive models convey limited utility for colleges. At the same time, recent research demonstrates that predictive models do not perform equally well for all student subgroups and that the specific modeling strategy used in predictive analytics can result in different student risk assignments (Bird, Castleman, Mabel, et al., 2021b). Additional research is needed to critically investigate whether the use of predictive analytics in higher education leads to more effective, efficient, and fair targeting of students for success‐oriented interventions.…”

Salient syllabi: Examining design characteristics of science online courses in higher education

“…In contrast to more resource-intensive course observations [66,67], course syllabi and institutional data are more readily available at colleges [68][69][70][71]. Therefore, this study can provide insights on how universities could apply syllabi-based rubrics to generate inferences for educational policies, for instance, when deciding what courses to keep in an online format after the Covid-19 pandemic [72], or in an effort to use predictive analytics to enhance learning outcomes [73][74][75]. Instead of traditional early warning systems that intend to identify at-risk students using institutional data during a college career and/or clickstream data during a specific course [74, 76, 77], syllabi-based early warning systems would identify courses not leveraging the affordances of online learning, and could be addressed even before students are exposed to the instructional enactments.…”

Education Data Science: Past, Present, Future

“…Often a simpler model does as well as a complex one, and usually, obvious factors play outsized roles (e.g., path dependence). The article exemplifying this in our special topic is written by Kelli Bird, Ben Castelman, Zachary Mabel, and Yifeng Song (Bird et al, 2021; “Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education”). Their article attempts to predict college dropouts using a variety of metrics and machine learning approaches and calls for greater transparency and critique of them.…”

“…In higher education, however, EDS methods are beginning to provide new complementary perspectives on institutional and student-level data (Chaturapruek et al, 2021). As these data become more readily available and joinable, machine learning can be used to synthesize and make comprehensible the rich contexts students traverse throughout their postsecondary paths (Pardos et al, 2019) and open up new avenues of intervention using predictive models (Bird et al, 2021). The introduction of data science approaches need not mean fields jettison well-earned advances and established tenets, that is, this is not a case of interfield colonization.…”

“…From our vantage, there should be a clear focus on problems of relevance to education that are potentially tractable given the data we have on hand. Even though there has been a relative explosion of data in education, we still have far less data (i.e., data rich with many fields) than is available in other domains, and this might limit the applicability of the most sophisticated algorithmic approaches (Bird et al, 2021). We should also work to build an ethic of responsibility in students.…”