Advanced, Analytic, Automated (AAA) Measurement of Engagement During Learning

Abstract

It is generally acknowledged that engagement plays a critical role in learning. Unfortunately, the study of engagement has been stymied by a lack of valid and efficient measures. We introduce the advanced, analytic, and automated (AAA) approach to measure engagement at fine-grained temporal resolutions. The AAA measurement approach is grounded in embodied theories of cognition and affect, which advocate a close coupling between thought and action. It uses machine-learned computational models to automatically infer mental states associated with engagement (e.g., interest, flow) from machine-readable behavioral and physiological signals (e.g., facial expressions, eye tracking, click-stream data) and from aspects of the environmental context. We present15 case studies that illustrate the potential of the AAA approach for measuring engagement in digital learning environments. We discuss strengths and weaknesses of the AAA approach, concluding that it has significant promise to catalyze engagement research.

An automated behavioral measure of mind wandering during computerized reading

“…The next challenge is to leverage them to build models that can detect mind wandering. One approach is to use supervised machine learning techniques (Domingos, 2012) to build a computational model of the relationship between a measure (in this case, eye gaze; see below) and instances of self-reported mind wandering (D'Mello, Duckworth, & Dieterle, 2017). The Blearned^model serves as a mind-wandering detector, using a machine-readable data source (e.g., eye gaze, neural activity) to reproduce a humanprovided one (e.g., self-reported mind wandering).…”

Observing response processes with eye tracking in international large-scale assessments: evidence from the OECD PIAAC assessment

“…As Oranje et al (2017) have argued that process data from log files and eye tracking observations can be integrated into the design and analysis of performance in 'next generation' assessment. It is quite possible that with the rapid pace of technological development, eye-tracking data may become a mainstream part of assessment practice (D'Mello et al, 2017). As this paper has demonstrated, eye tracking data enables researchers to dig deeper into response processes and to investigate, profile and explain test-taker performance.…”

“…However, the rise of research into 'processes data' in assessment (Ercikan and Pellegrino 2017;Zumbo and Hubley 2017) and advances in eye tracker technology (Bixler and D'Mello 2016;D'Mello et al, 2017) suggest that the application of eye tracking techniques can be extended to observations of how tested populations receive and engage with test items. The contribution of eye tracking in large-scale assessments relates to at least two distinct areas.…”

“…log files, eye tracking, emotion recognition, linguistic data and think aloud protocols) to inform our understanding of item response processes (e.g. D'Mello et al, 2017;Maddox and Zumbo 2017;Ercikan and Pellegrino 2017;Zumbo and Hubley 2017).…”

“…As eye tracking and gaming technology advances, eye tracking is likely to be incorporated into computers, tablets and smart phones as standard equipment. This will increase the potential for eye tracking to get out of the lab, to inform analysis of respondent performance (see D'Mello et al, 2017). Desk-mounted eye trackers (such as the EyeLink 1000) offer high levels of precision and speed (i.e.…”