Leveraging process data to assess adults’ problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks

He, Qiwei; Borgonovi, Francesca; Paccagnella, Marco

doi:10.1016/j.compedu.2021.104170

Cited by 60 publications

(62 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These remain to be investigated and understood in future research. Second, in line with previous studies (He, Borgonovi, & Paccagnella, 2019, 2021He, Liao, & Jiao, 2019;Tang et al, 2020a), the fact that features typically encountered in incorrect action sequences tended to be rather heterogeneous indicates that different sources of failure, each describable by different features, tend to underlie incorrect responses.…”

Section: Differences Between Incorrect and Correct Behavioral Patternssupporting

confidence: 80%

“…Contrasting incorrect against correct behavioral patterns. Commonly, studies exploring how examinees approach interactive tasks have found action sequences and time spent on task of incorrect responses to display considerably larger variation than those of correct responses (He, Borgonovi, & Paccagnella, 2019, 2021He, Liao, & Jiao, 2019;He & von Davier, 2016;Stadler et al, 2019;Tang et al, 2020a). These findings can be interpreted as an indicator of the greater variety of behavioral patterns associated with incorrect as compared to correct responses.…”

Section: Previous Research On Incorrect Responses To Interactive Tasksmentioning

confidence: 99%

“…With further theoretical advances, we also expect theories to cover the specific characteristics of incorrect response behavior on interactive tasks. To corroborate theories on incorrect response behavior, expert-defined incorrect and correct behavioral patterns may be compared to observed ones (as in He, Borgonovi, & Paccagnella, 2019, 2021. One of the main challenges, however, is the integration of timing information into such comparisons.…”

Section: Limitations and Future Researchmentioning

confidence: 99%

See 2 more Smart Citations

Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks

Ulitzsch

Pohl

2021

Journal of Educational and Behavioral Statistics

Self Cite

View full text Add to dashboard Cite

Interactive tasks designed to elicit real-life problem-solving behavior are rapidly becoming more widely used in educational assessment. Incorrect responses to such tasks can occur for a variety of different reasons such as low proficiency levels, low metacognitive strategies, or motivational issues. We demonstrate how behavioral patterns associated with incorrect responses can, in part, be understood, supporting insights into the different sources of failure on a task. To this end, we make use of sequence mining techniques that leverage the information contained in time-stamped action sequences commonly logged in assessments with interactive tasks for (a) investigating what distinguishes incorrect behavioral patterns from correct ones and (b) identifying subgroups of examinees with similar incorrect behavioral patterns. Analyzing a task from the Programme for the International Assessment of Adult Competencies 2012 assessment, we find incorrect behavioral patterns to be more heterogeneous than correct ones. We identify multiple subgroups of incorrect behavioral patterns, which point toward different levels of effort and lack of different subskills needed for solving the task. Albeit focusing on a single task, meaningful patterns of major differences in how examinees approach a given task that generalize across multiple tasks are uncovered. Implications for the construction and analysis of interactive tasks as well as the design of interventions for complex problem-solving skills are derived.

show abstract

Section: Differences Between Incorrect and Correct Behavioral Patternssupporting

confidence: 80%

Section: Previous Research On Incorrect Responses To Interactive Tasksmentioning

confidence: 99%

Section: Limitations and Future Researchmentioning

confidence: 99%

See 1 more Smart Citation

Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks

Ulitzsch

Pohl

2021

Journal of Educational and Behavioral Statistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…The information extracted from process data is particularly valuable when examining interactive problem‐solving tasks (He et al, 2019, 2021). The analysis of process data has recently gained strong interests, with a wide range of model‐ and data‐ driven methods proposed to understand the types of strategies that contribute to successful and unsuccessful problem‐solving, identify the behavioural differences between observed and latent subgroups and assess the proficiency on the trait of interests (e.g., Goldhammer et al, 2013; He et al, 2019, 2021; He & von Davier, 2016; LaMar, 2018; Tang et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

“…In the first category of sequence‐based analysis, distances between two or multiple sequences are often computed. For instance, He et al (2019, 2021) applied the longest common subsequence method (Hirschberg, 1975, 1977; Sukkarieh et al, 2012) to identify how different the action sequence taken by a test taker to solve an item is from the action sequence that subject matter experts consider optimal to solve the problem. The longer the distance from the predefined action sequences, the less likelihood of a correct response.…”

Section: Introductionmentioning

confidence: 99%

Exploring latent states of problem‐solving competence using hidden Markov model on process data

Xiao

Veldkamp

et al. 2021

Computer Assisted Learning

Self Cite

View full text Add to dashboard Cite

The response process of problem‐solving items contains rich information about respondents' behaviours and cognitive process in the digital tasks, while the information extraction is a big challenge. The aim of the study is to use a data‐driven approach to explore the latent states and state transitions underlying problem‐solving process to reflect test‐takers' behavioural patterns, and to investigate how these states and state transitions could be associated with test‐takers' performance. We employed the Hidden Markov Modelling approach to identify test takers' hidden states during the problem‐solving process and compared the frequency of states and/or state transitions between different performance groups. We conducted comparable studies in two problem‐solving items with a focus on the US sample that was collected in PIAAC 2012, and examined the correlation between those frequencies from two items. Latent states and transitions between them underlying the problem‐solving process were identified and found significantly different by performance groups. The groups with correct responses in both items were found more engaged in tasks and more often to use efficient tools to solve problems, while the group with incorrect responses was found more likely to use shorter action sequences and exhibit hesitative behaviours. Consistent behavioural patterns were identified across items. This study demonstrates the value of data‐driven based HMM approach to better understand respondents' behavioural patterns and cognitive transmissions underneath the observable action sequences in complex problem‐solving tasks.

show abstract

Process Data Analysis in ILSAs

Costa

Netto²

2022

International Handbook of Comparative Large-Scale Studies in Education

View full text Add to dashboard Cite

Computational advancements in the last couple of decades have brought forth a new era of international large-scale assessments (ILSAs) in which the administration of computer-based tests is becoming the norm. Beyond collecting the correct/incorrect answers for each item, computerbased assessments are also able to collect a range of actions performed by the respondents in the computer testing application during the course of the test administration. Both respondents' actions-starting a unit, clicking a button, spending time until inputting or submitting an answer, and so forth-and their overall behavior-what they do with their keyboards, mice, and even their own eyes-are recorded as a new set of data called "process data", which are normally timestamped and can be stored in so-called log files. There may be plenty of useful insight into the respondent's cognitive process, and process data can potentially become a relevant element in the scoring process of an assessment, validate test score interpretations, to name a few possibilities for the analysis of such data. This chapter aims to contribute to the body of knowledge in this area by offering (1) an introduction to process data, what kind of data it contains, its relation to the cognitive process and how it can be organized into a proposed 6-layered ecological framework that facilitates its analysis; (2) a literature review of 37 seminal and state-of-the-art publications produced in the last decade on the topic, which are then analyzed both in their chronological perspective as well as in how they fit into the ecological framework; and (3) a discussion of the potential and limitations of using process data in the assessment framework, including a view of what could be the next steps in the analysis of process data from ILSAs.

show abstract

Leveraging process data to assess adults’ problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks

Cited by 60 publications

References 52 publications

Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks

Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks

Exploring latent states of problem‐solving competence using hidden Markov model on process data

Process Data Analysis in ILSAs

Contact Info

Product

Resources

About