Towards investigating the validity of measurement of self-regulated learning based on trace data

Abstract

Contemporary research that looks at self-regulated learning (SRL) as processes of learning events derived from trace data has attracted increasing interest over the past decade. However, limited research has been conducted that looks into the validity of trace-based measurement protocols. In order to fill this gap in the literature, we propose a novel validation approach that combines theory-driven and data-driven perspectives to increase the validity of interpretations of SRL processes extracted from trace-data. The main contribution of this approach consists of three alignments between trace data and think aloud data to improve measurement validity. In addition, we define the match rate between SRL processes extracted from trace data and think aloud as a quantitative indicator together with other three indicators (sensitivity, specificity and trace coverage), to evaluate the “degree” of validity. We tested this validation approach in a laboratory study that involved 44 learners who learned individually about the topic of artificial intelligence in education with the use of a technology-enhanced learning environment for 45 minutes. Following this new validation approach, we achieved an improved match rate between SRL processes extracted from trace-data and think aloud data (training set: 54.24%; testing set: 55.09%) compared to the match rate before applying the validation approach (training set: 38.97%; test set: 34.54%). By considering think aloud data as “reference point”, this improvement of the match rate quantified the extent to which validity can be improved by using our validation approach. In conclusion, the novel validation approach presented in this study used both empirical evidence from think aloud data and rationale from our theoretical framework of SRL, which now, allows testing and improvement of the validity of trace-based SRL measurements.

Improving the measurement of self-regulated learning using multi-channel data

“…Samuel Messick defined validity as the "integrated evaluative judgement of degree to which empirical evidence and theoretical rationale supports the adequacy and appropriateness of inferences and actions based on test scores" (Messick, 1994, p. 6). Therefore, in this study, we followed a validation approach that combines theory-driven and data-driven perspectives to ensure the validity of interpretations of SRL processes extracted from tracedata (Fan et al, 2022).…”

“…In order to improve the validity of our trace-based measurement protocol, we also collected think aloud data and used it to triangulate with trace data. Although think aloud method has its own limitations (Young, 2005;Fan et al, 2022), the interpretations based on think aloud codes are still considered a relatively more valid measurement than the use of self-report survey data (Bannert and Mengelkamp, 2013;Veenman, 2007;Bannert, 2007;Greene & Azevedo, 2010;Fan et al, 2022). Therefore, we considered think aloud codes as "reference points" in this study to test the validity of the trace-based measurement protocol.…”

“…This results indicated that the interpretations in our trace-based measurement protocol are valid to a certain extent when considering think aloud as a "reference point". The full description of our validation process could be found in Fan et al (2022).…”

“…As shown in this study, cross-validation across multiple data channels can itself enhance the reliability of the rules for detection of SRL processes and the validity of measurements. Another approach to improving the validity is by using think aloud data to validate and triangulate inferences drawn from the data channels analysed in the current study (Fan et al, 2022).…”

Analysis of the Deep Learning Strategies Questionnaire with Ecuadorian students

“…Measuring self-regulative learning strategies is still challenging for researchers (Veenman, 2007;Fan et al, 2022). One way to assess self-regulation may be to identify students' strategies in realistic situations.…”