Learned student models with item to item knowledge structures

Abstract

Abstract. Probabilistic and learned approaches to student modeling are attractive because of the uncertainty surrounding the student skills assessment and because of the need to automatize the process. Item to item structures readily lend themselves to probabilistic and fully learned models because they are solely composed of observable nodes, like answers to test questions. Their structure is also well grounded in the cognitive theory of knowledge spaces. We study the effectiveness of two Bayesian frameworks to learn item to item structures and to use the induced structures to predict item outcome from a subset of evidence. One approach, POKS, relies on a naive Bayes framework whereas the other is based on the Bayesian network learning and inference framework. Both approaches are assessed over their predictive ability and their computational efficiency in different experimental simulations. The results from simulations over three data sets show that they both can effectively perform accurate predictions, but POKS generally displays higher predictive power than the Bayesian network. Moreover, the simplicity of POKS translates to a time efficiency of one to three orders of magnitude greater than the Bayesian network runs. We furhter explore the use of the item to item approach for handling concepts mastery assessment. The approach investigated consist in augmenting an initial set of observations, based on inferences with the item to item structure, and feed the augmented set to a Bayesian network containing a number of concepts. The results show that augmented set can effectively improve predictive power of a Bayesian network for item outcome, but that improvement does not transfer to the concept assessment in this particular experiment. We discuss different explanations for the results and outline future research avenues.

Performance Comparison of Item-to-Item Skills Models with the IRT Single Latent Trait Model

“…This would not the case if we wanted to predict the mastery of a set of (unobservable) skills, in which case both approaches would require some knowledge engineering eort, such as dening a Q-matrix or dening the topology of a Bayesian Network, as well as independent means to assess the skills for validation purpose. Similar studies were conducted by Desmarais et al, [6,7]. These studies respectively compared the performance of a Bayesian Network developed by Vomlel [23] and of the IRT approach with a derivative of a Naive Bayes item-to-item model (POKS).…”

“…Bayesian Network) is a compromise between a number of factors to consider, such as knowledge engineering eorts, computational complexity, and most importantly reliability and accuracy of predictions. A number of researchers in the learner modeling eld have investigated this issue over the last decade or so [22,5,4,6,1,15]. This paper revisits the issue of assessing item-to-item model performance by comparing the predictive accuracy of standard Bayesian classier algorithms [10] to create item-to-item learner models with that of the IRT approach (see [21]), which contains a single latent skills.…”

“…Because the POKS technique used in the work of Desmarais et al (2005Desmarais et al ( , 2006 integrates a feature selection algorithm in addition to the probabilistic inference techniques listed, it cannot be directly compared here. However, given that POKS uses a Naive Bayes inference rule, the performance would be expected to be the same as the Naive Bayes technique of this study.…”

A review of recent advances in learner and skill modeling in intelligent learning environments

“…ALEKS is a commercial spin off of the University of California at Irvine and is based on the cognitive theory of knowledge spaces Falmagne 1999, 1985). This theory is at the basis of a number of efforts and active developments in the field of learner modeling (Heller et al 2006;Desmarais et al 2006).…”

Network Medicine: New Paradigm in the -Omics Era

“…As an alternative to Bayesian networks, an implication network model employs a partial order knowledge structure (POKS) for structural learning and uses the Bayesian theory for inference propagation [201,202]. When using Dempster-Shafer theory for belief updating, this implication network methodology is termed a DempsterShafer belief network [203,204].…”