Large-scale evaluation of cold start mitigation in adaptive fact learning: Knowing "what" matters more than knowing "who"

Abstract

Adaptive learning systems offer a personalised digital environment that continually adjusts to the learner and the material, with the goal of maximising learning gains. Whenever such a system encounters a new learner, or when a returning learner starts studying new material, the system first has to determine the difficulty of the material for that specific learner. Failing to address this "cold start" problem leads to suboptimal learning and potential disengagement from the system, as the system may present problems of an inappropriate difficulty or provide unhelpful feedback. In a simulation study conducted on a large educational data set from an adaptive fact learning system (about 100 million trials from almost 140 thousand learners), we predicted individual learning parameters from response data. Using these predicted parameters as starting estimates for the adaptive learning system yielded a more accurate model of learners' memory performance than using default values. We found that predictions based on the difficulty of the fact ("what") generally outperformed predictions based on the ability of the learner ("who"), though both contributed to better model estimates. These findings confirm the benefit of fact-specific predictions that we previously found in a smaller-scale lab-based experiment, while also showing that learner-specific predictions can aid cold start mitigation in a real-world educational setting. The improved predictions can be harnessed to increase the efficiency of the learning system, mitigate the negative effects of a cold start, and potentially improve learning outcomes.