Bayes Nets in Educational Assessment: Where Do the Numbers Come From?

Mislevy, Robert J.; Almond, Russell G.; Yan, Duanli; Steinberg, Lynne

doi:10.1037/e649912011-001

Cited by 54 publications

(78 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such hierarchies are depicted in Chapter 4 of Rupp, Templin, and Henson (2010) and have been anticipated in many other context, both in diagnostic psychometric models (see Leighton, Gierl, & Hunka, 2004;Tatsuoka, 2002Tatsuoka, , 1983 and in machine learning contexts (see Mislevy, Almond, & Yan, 1999;Pardos, Heffernan, Anderson, Heffernan, & Schools, 2010). The structure of the HDCM allows for its use beyond linear hierarchies, by which we believe the value of the HDCM is not only self-evident but is actually consistent with the values of parsimony espoused by von Davier and Haberman (2014) in their commentary.…”

Section: In Addition Under the Assumption Of Attribute Hierarchies mentioning

confidence: 62%

The Use and Misuse of Psychometric Models

Templin

Bradshaw

2014

Psychometrika

View full text Add to dashboard Cite

The purpose of our paper entitled Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies was two-fold: to create a psychometric model and framework that would enable attribute hierarchies to be parameterized as dependent binary latent traits, and to formulate an empirically driven hypothesis test for the purpose of falsifying proposed attribute hierarchies. The methodological contributions of this paper were motivated by a curious result in the analysis of a real data set using the log-linear cognitive diagnosis model, or LCDM (Henson, Templin, & Willse, 2009). In the analysis of the Examination for Certification of Proficiency in English (ECPE; Templin & Hoffman, 2013), results indicated that few, if any, examinees were classified into four of the possible eight attribute profiles that are hypothesized in the LCDM for a test of three binary latent attributes. Further, when considering the four profiles lacking examinees, it appeared that some attributes must be mastered before others, suggesting what is commonly called an attribute hierarchy (e.g., Leighton, Gierl, & Hunka, 2004). Although the data analysis alerted us to the notion that such a data structure might be present, we lacked the methodological tools to falsify the presence of such an attribute hierarchy. As such, we developed the Hierarchical Diagnostic Classification Model, or HDCM, in an attempt to fill the need for such tools.We note that the driving force behind the HDCM is one of seeking a simpler, or more parsimonious, solution when model data misfit is either evident from LCDM results or implied by the hypothesized theories underlying the assessed constructs. As a consequence of the ECPE data results, we worked to develop a more broadly defined set of models that would allow for empirical evaluation of hypothesized attribute hierarchies. We felt our work was timely, as a number of methods, both new and old, are now using implied attribute hierarchies to assess examinees in many large scale analyses-from so-called intelligent tutoring systems (e.g

show abstract

Section: In Addition Under the Assumption Of Attribute Hierarchies mentioning

confidence: 62%

The Use and Misuse of Psychometric Models

Templin

Bradshaw

2014

Psychometrika

View full text Add to dashboard Cite

show abstract

“…In future administrations, the population distribution can be linked by including a set of common tasks in both administrations (this is a Bayesian network version of the Non-Equivalent groups Anchor Test, NEAT, design; Mislevy et al, 1999). In this case, the prior distribution for the link model parameters for the common task is taken as the posterior distribution from the previous administration.…”

Section: Proficiency Tasks In Anchor (3 Observables Each)mentioning

confidence: 99%

Parameter Recovery Studies With a Diagnostic Bayesian Network Model

2008

Self Cite

View full text Add to dashboard Cite

This paper describes a Bayesian network model for a candidate assessment design that had four proficiency variables and 48 tasks with 3-12 observable outcome variables per task and scale anchors to identify the location of the subscales. The domain experts' view of the relationship among proficiencies and tasks established a complex prior distribution over 585 parameters. Markov Chain Monte Carlo (MCMC) estimation recovered the parameters of data simulated from the expert model. The sample size and the strength of the prior had only a modest effect on parameter recovery, but did affect the standard error of estimated parameters. Finally, an identifiability issue involving relabeling of proficiency states and permutations of the matrixes is addressed in the context of this study. Scope of StudiesBayesian networks form an attractive class of models for diagnostic assessments as they can relate multiple proficiency variables to multiple dependent observables coming from 1. There were multiple correlated proficiencies, which must be simultaneously estimated. 2. The observable outcome variables were grouped into tasks of varying complexity; each task tapped one or more proficiencies. 3. Tasks had different target difficulties. 4. The tasks were to be presented in complex balanced incomplete block design to ensure sufficient concentration around each of seven potential proficiencies.This paper describes simulation studies designed to answer the question of "Can an assessment with these characteristics be modeled with a Bayesian network?" assuming that there was no local dependence between observables in the same task. This paper describes a number of parameter recovery experiments based on the assessment design described in Section 2. The methodology was roughly similar for all of the experiments. In particular,Key Words and Phrases: Bayesian networks, model construction, Markov Chain Monte Carlo (MCMC) estimation, prior elicitation, scale anchor * Educational Testing Service, MS 13-E Princeton, NJ 08541 USA Mail Address: ralmond@ets.org, almond@acm.org 1) Later versions of this assessment were named iSkills TM . Note that the work described here is based on a very preliminary model of the ICT Literacy which has undergone considerable refinement since the studies described here were carried out and that the models described in this paper do not reflect current thinking about this construct or the scoring methods used in the iSkills assessment.

show abstract

“…There is considerable interest in the use of Bayesian networks (BN) for student modeling and cognitive assessment (for eg., Mislevy, Almond, Yan, & Steinberg, 1999;Conati, Gertner, & VanLehn, 2002;VanLehn, Niu, Siler, & Gertner, 1998;Vomlel, 2004). In part, this interest stems from the ability of a BN to model the uncertainty that is inherent to cognitive modeling.…”

Section: Introductionmentioning

confidence: 99%

Bayesian Student Models Based on Item to Item Knowledge Structures

Desmarais

Gagnon

2006

Innovative Approaches for Learning and Knowledge Sharing

View full text Add to dashboard Cite

Abstract. Bayesian networks are commonly used in cognitive student modeling and assessment. They typically represent the item-concepts relationships, where items are observable responses to questions or exercises and concepts represent latent traits and skills. Bayesian networks can also represent concepts-concepts and concepts-misconceptions relationships. We explore their use for modeling item-item relationships, in accordance with the theory of knowledge spaces. We compare two Bayesian frameworks for that purpose, a standard Bayesian network approach and a more constrained framework that relies on a local independence assumption. Their performance is compared over their respective ability to predict item outcome and through simulations over two data sets. The simulation results show that both approaches can effectively perform accurate predictions, but the constrained approach shows higher predictive power than a Bayesian Network. We discuss the applications of item to item structure for cognitive modeling within different contexts.

show abstract

Bayes Nets in Educational Assessment: Where Do the Numbers Come From?

Cited by 54 publications

References 14 publications

The Use and Misuse of Psychometric Models

The Use and Misuse of Psychometric Models

Parameter Recovery Studies With a Diagnostic Bayesian Network Model

Bayesian Student Models Based on Item to Item Knowledge Structures

Contact Info

Product

Resources

About