A Diagnostic Classification Analysis of Problem-Solving Competence using Process Data: An Item Expansion Method

Abstract

Process data refers to data recorded by computer-based assessments (CBA) that reflect respondents’ problem-solving processes and provide greater insight into how students solve problems, instead of merely how well they solve them. Using the rich information contained in process data, this study proposed an item-specific psychometric method for analyzing process data in order to comprehensively understand respondents’ problem-solving competence. By incorporating diagnostic classification into process data analysis, the proposed method cannot only estimate respondents’ problem-solving ability along a continuum, but can also classify respondents according to their problem-solving strategies. To illustrate the application and advantages of the proposed method, a Programme for International Student Assessment (PISA) problem-solving task was used. The results indicated that (a) the estimated latent classes provided more detailed diagnoses of respondents’ problem-solving strategies than the observed score classes; (b) although only one item was used, estimated higher-order latent ability reflected the respondents’ problem-solving ability more accurately than the estimated unidimensional latent ability taken from the outcome data; and (c) the interactions between problem-solving skills may follow the conjunctive condensation rule, which assumes that only when a respondent has mastered all the required problem-solving skills can the specific action sequence appear. Overall, the main conclusion drawn from this study was that using diagnostic classification is a feasible and promising method for analyzing process data.

Joint Modeling of Action Sequences and Action Times in Problem-solving Tasks

“…For participants with different levels of problem-solving competence, the length and sequence of actions or states may be different. Thus, action sequences or the problem state sequence they comprise can be used as the observed performance to infer the latent problem-solving ability of the participants (Han et al, 2021;Zhan & Qiao, 2022). Essentially, the transition between two adjacent stages to solve the problem is similar to…”

“…Feature extraction includes both theory-driven and data-driven approaches. The former mainly uses expert-defined behavioral indicators to rate (Harding et al, 2017;Rosen, 2017;Yuan et al, 2019) or recode the action sequences of participants (Liu et al, 2018;Zhan & Qiao, 2022;Zhang et al, 2022), which, however, requires different feature extraction rules for different tasks, making its application dependent on a considerable amount of expert judgment. The latter refers to the application of data mining, machine learning, and other methods to directly extract information from process data, such as natural language processing (Hao et al, 2015;He et al, 2021;He & von Davier, 2016), dimensionality reduction algorithms (Tang et al, 2016;Tang et al, 2020;Tang et al, 2021), and network analysis (Zhu et al, 2016).…”

“…It includes the application of traditional psychometric models and stochastic process modeling. The former focuses on how to analyze the extracted and recoded behavioral indicators using traditional models, such as item response theory (IRT) and cognitive diagnostic models (Liu et al, 2018;Wilson et al, 2017;Zhan & Qiao, 2022;Zhang et al, 2022;Zhu et al, 2016), while the latter considers how to model action sequences directly via, for example, the dynamic Bayesian network (Levy, 2019) and hidden Markov model (Arieli-Attali et al, 2019;Bergner et al, 2017).…”

Does Diagnostic Feedback Promote Learning? Evidence From a Longitudinal Cognitive Diagnostic Assessment

“…Li et al, 2020), spatial rotation (S. Wang et al, 2018), problem-solving competence (Zhan & Qiao, 2020), situational judgment (Sorrel et al, 2016), and psychological disorders (Templin & Henson, 2006).…”

“…CDAs offer researchers a framework of formative assessments that are capable of providing educators with detailed information about student mastery status (e.g., mastery or nonmastery) of latent attributes in a given subject area, with examples including (a) fraction subtraction (Tatsuoka, 1983), (b) rational number operations , (c) voltage and current (Ohm's law) (Zhan et al, 2019), (d) geometric sequences (Shute et al, 2008), (e) buoyancy (Gao et al, 2020), and (f) proportional reasoning (Tjoe & de la Torre, 2013). Researchers have applied CDAs more broadly in the psychological and behavioral sciences, such as for learning mode (H. Li et al, 2020), spatial rotation (S. Wang et al, 2018), problem-solving competence (Zhan & Qiao, 2020), situational judgment (Sorrel et al, 2016), and psychological disorders (Templin & Henson, 2006).…”