Testing Theories of Transfer Using Error Rate Learning Curves

Abstract

We analyze naturally occurring datasets from student use of educational technologies to explore a long-standing question of the scope of transfer of learning. We contrast a faculty theory of broad transfer with a component theory of more constrained transfer. To test these theories, we develop statistical models of them. These models use latent variables to represent mental functions that are changed while learning to cause a reduction in error rates for new tasks. Strong versions of these models provide a common explanation for the variance in task difficulty and transfer. Weak versions decouple difficulty and transfer explanations by describing task difficulty with parameters for each unique task. We evaluate these models in terms of both their prediction accuracy on held-out data and their power in explaining task difficulty and learning transfer. In comparisons across eight datasets, we find that the component models provide both better predictions and better explanations than the faculty models. Weak model variations tend to improve generalization across students, but hurt generalization across items and make a sacrifice to explanatory power. More generally, the approach could be used to identify malleable components of cognitive functions, such as spatial reasoning or executive functions.

Discovering Psychological Principles by Mining Naturally Occurring Data Sets

“…Another obvious difference in emphasis between CNPs and traditional ecological psychologists is that the former are focusing on the structure of an external world that is largely based on information on the web or social constructs—words (Christiansen & Monaghan; Vincent‐Lamarre), prices (Pope), stocks (Moat et al. ), mathematics problems (Koedinger, Yudelson, & Pavlik, ), and scholarly articles (Berger). As more of our attended world migrates online, it makes sense that the structure of this online world becomes of central psychological concern.…”

“…Koedinger et al. () show how the choice between different cognitive models of mathematical reasoning can be adjudicated with the help of large amounts of data, courtesy of the Datashop repository of educational technology data (Koedinger, Stamper, Leber, & Skogsholm, ). The particular data they studied concerned the patterns of successes and errors that students make across a suite of related mathematical problems.…”

“…Koedinger et al. () describe another nuisance factor in their NODS; because they use an adaptive mastery algorithm, the number of students contributing to an error rate estimate decreases as the number of practice opportunities increases. Braithwaite, Goldstone, van der Maas, and Landy () describe a related measurement problem with educational systems that adapt the problems given to students based on the students’ performance.…”

Data mining and education

“…All the BKT models and the newer logistic regression models model learning by including estimates of learning for each KC. These models use knowledge components as the basis for explaining learning, though it is possible to have a descriptive model of learning (not shown in Table ) that would have a single learning parameter across all tasks, as in a faculty theory of transfer (cf.,).…”

“…For example, estimations of learning and item difficulty can be confounded by sampling issues. A lack of randomization of items in a curriculum might be responsible for performance variance due to learning being attributed to item parameters (cf.,). For example, an item that is consistently completed at the end of a unit is more likely to benefit from transfer of learning from earlier items than an item that is randomly distributed throughout the unit.…”