Bayesian Network Models for Local Dependence Among Observable Outcome Variables

Abstract

Bayesian network models offer a large degree of flexibility for modeling dependence among observables (item outcome variables) from the same task, which may be dependent. This article explores four design patterns for modeling locally dependent observations: (a) no context-ignores dependence among observables; (b) compensatory context-introduces a latent variable, context, to model task-specific knowledge and use a compensatory model to combine this with the relevant proficiencies; (c) inhibitor context-introduces a latent variable, context, to model task-specific knowledge and use an inhibitor (threshold) model to combine this with the relevant proficiencies; (d) compensatory cascading-models each observable as dependent on the previous one in sequence. This article explores the four design patterns through experiments with simulated and real data. When the proficiency variable is categorical, a simple Mantel-Haenszel procedure can test for local dependence. Although local dependence can cause problems in the calibration, if the models based on these design patterns are successfully calibrated to data, all the design patterns appear to provide very similar inferences about the Joris Mulder is now with Utrecht University. Sections 3, 4, and 5 of this article are based on the work that Joris Mulder did while a summer intern at ETS. An expanded description of those experiments is available in his final report. We thank the team doing the parallel IRT analysis, including Brad Moulder, Frank Jenkins, Bruce Kaplan, Quncai Zhou, and Youn-Hee Lim, for generously sharing their results with us. Peggy Redman and Irv Katz helped by providing us with lots of the ECD documentation for the tasks used in the ICT literacy assessment. Alexander Matukhin did much of the programming for the StatShop software used in this analysis. Bob Mislevy and David Williamson provided helpful suggestions and cheerful encouragement throughout the project. Dan Eignor and Lou DiBello provided a number of suggestions that were quite helpful in improving the clarity of the presentation. Thanks also to David Rindskopf for finding two anonymous reviewers whose questions and insights helped improve the current article. Downloaded from students. Based on these experiments, the simpler no context design pattern appears more stable than the compensatory context model, while not significantly affecting the classification accuracy of the assessment. The cascading design pattern seems to pick up on dependencies missed by other models and should be explored with further research.

Hybrid Measurement Models for Technology-Enhanced Assessments Through mIRT-bayes

“…While a two-stage process is used here, extension to one-stage is also possible (Almond et al, 2009). However, there are advantages in flexibility of assessment development and scoring to keep the analytic process as two-stage.…”

“…Bayes Nets are also well understood models previously presented extensively in the research literature as stand-alone models and used for instance in classroom-based assessments (Almond, DiBello, Moulder, & Zapata-Rivera, 2007;Almond, Mulder, Hemat, & Yan, 2009 Bayes Nets are acyclic directed graph structures, represented with nodes and arcs, which are shown in the Results section in an example. To generate the joint probabilities in the network, a strong assumption is made in Bayesian Network approaches that P, the local probability distribution of variable X i , is made conditional only on the value of the parent nodes connecting to a given node from above.…”

“…One approach could draw on Bayesian network models with continuous proficiency variables as equivalent to multidimensional IRT models (Almond, Mulder, Hemat, & Yan, 2009). Where such configurations apply, it could be possible to institute design matrices or other approaches through the estimation software that sufficiently constrain portions of the calibration, such that the nested directed acyclic graph portion of the model and the overarching IRT structure are calibrated simultaneously, meaning within one estimation.…”

Computerized adaptive testing and learning using bayesian network

“…As shown in figure 1, all items belong to their associated categories (internal nodes of Bayesian network), and categorical connections can be modeled as cause-effect relationships of Bayesian network [2]. This modeling may be very useful to improve accuracy of learner's proficiency estimation because it reflects dependency information between items on the test of learner.…”

Research on Statistics

“…The model is described in Mislevy and Gitomer (1995), building on previous work by Mislevy (1994a, b). Further developments can be found in Almond et al (2009).…”