Developing and Validating Cognitive Models in Assessment

Integrating Multiple Sources of Validity Evidence for an Assessment‐Based Cognitive Model

“…Analyzing TCE as a pupillary measure of cognitive effort for each item yielded valuable information linking patterns of cognitive effort and correct or incorrect item responses. Additionally, it provided evidence supporting the proposed skills‐based cognitive model (Keehner et al., ; Slaney, ). The Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, ) encourage the evaluation of evidence related to cognitive processes as a part of assessment validation; as a result, data evaluating the relationship of cognitive effort to item difficulty provided support for the model and provides crucial validation evidence.…”

“…Designing an assessment by first articulating the cognitive model that explicitly defines the knowledge and skills to be measured has several advantages (Embretson, ; Keehner et al., ). Whereas traditional assessment design methods contribute limited understanding regarding test takers’ thinking related to problem solving, by specifying the cognitive model a direct link between item/task performance and knowledge or skill level is established (Embretson & Gorin, ; National Research Council, ).…”

“…Leighton and Gierl () defined a cognitive model as a “simplified description of human problem solving on standardized educational tasks, which helps to characterize the knowledge and skills students at different levels of learning have acquired and to facilitate the explanation and prediction of students’ performance” (p. 6). Cognitive models attempt to describe the cognitive processes involved in decoding information, retrieving knowledge, problem solving, and decision making (Keehner, Gorin, Feng, & Katz, ). When applied to problems or tasks that constitute educational or psychological assessments, cognitive models represent the formalized processes by which an examinee decodes the task presented by an item, works to solve the problem, and finally provides a response based on the work.…”

Development of a novel hybrid cognitive model validation framework for implementation under COVID‐19 restrictions

“…Keehner et al (2017) discuss the general approach to validation of cognitive models, highlighting the requirement for an iterative, staged process. The goal of any validation method should be to support claims and validity arguments about the specific models to which it is applied (Keehner et al, 2017).…”

“…Keehner et al (2017) discuss the general approach to validation of cognitive models, highlighting the requirement for an iterative, staged process. The goal of any validation method should be to support claims and validity arguments about the specific models to which it is applied (Keehner et al, 2017). Similarly, Kane (2013) argues that validation is an ongoing and iterative process, and we believe a structured validation framework could have utility in the early stages of research where gathering appropriate resources is difficult or not cost‐effective.…”