Tracking Patterns in Self-Regulated Learning Using Students’ Self-Reports and Online Trace Data

Abstract

For decades, self-report instruments-which rely heavily on students' perceptions and beliefs-have been the dominant way of measuring motivation and strategy use. Eventbased measures based on online trace data arguably has the potential to remove analytical restrictions of self-report measures. The purpose of this study is therefore to triangulate constructs suggested in theory and measured using self-reported data with revealed online traces of learning behaviour. The results show that online trace data of learning behaviour are complementary to self-reports, as they explained a unique proportion of variance in student academic performance. The results also reveal that self-reports explain more variance in online learning behaviour of prior weeks than variance in learning behaviour in succeeding weeks. Student motivation is, however, to a lesser extent captured with online trace data, likely because of its covert nature. In that respect, it is of importance to recognize the crucial role of self-reports in capturing student learning holistically. This manuscript is 'frontline' in the sense that event-based measurement methodologies with online trace data are relatively unexplored. The comparison with self-report data made in this manuscript sheds new light on the added values of innovative and traditional methods of measuring motivation and strategy use.

Self-Report is Indispensable to Assess Students’ Learning

“…These findings are consistent with the specificity matching principle (see, e.g., Swann et al, 2007): Variables show stronger relations when being matched in terms of situational specificity than when not being matched, and the present results suggest that self-report measures can demonstrate validity when attending to this principle. Van Halem et al (2020) used the Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich et al, 1991) as well as online trace data to assess undergraduate students' motivation and learning strategies during a statistics course. There was no direct conceptual match between the MSLQ and online trace data constructs.…”

A Proposed Remedy for Grievances about Self-Report Methodologies

“…These researchers also gathered participant's think aloud accounts about how participants studied an informative text. van Halem, et al, (2020) administered the well-known Motivated Strategies for Learning Questionnaire which includes various subscales of Likert items describing constructs within the arenas of motivation, cognition and metacognition. Participants in Vriesema and McCaslin's (2020) study responded to a survey including Likert response items about anxiety and selected from among a set of 20 sentences ones that described perceptions about participation in a small group activity.…”

“…On the other side of this model, every self report methodology specifies conditions, IFs, the context within which the learner is to reply. As noted earlier, these may be set out for the learner in general terms, e.g., "this course" as in the Motivated Strategies for Learning Questionnaire (see van Halem et al, 2020) or "the lecture contents of the past couple of minutes" (Moeller et al, 2020, p. 6). When a learner responds, it is reasonable to infer some particular features of a setting, IFs, influence the learner's response.…”

“…Second, it is often the case that self-report data are among other predictors used to predict an outcome. Multiple regression and similar models are common statistical methods applied in this case, as was true for studies in this special issue (Chauliac et al, 2020;Durik & Jenkins, 2020;van Halem et al, 2020;Moeller et al, 2020;Vriesema & McCaslin, 2020). In these types of analysis, each predictor is residualized for every other predictor.…”

Measurement and the Study of Motivation and Strategy Use: Determining If and When Self-report Measures are Appropriate

“…Ultimately, these data sources were combined to shed light on how self-reported individual differences related to SRL processes in either individual (Rogiers et al, 2020) or group (Vriesema & McCaslin, 2020) settings. In addition to these two studies, articles in this special issue show other ways of combining self-report survey responses with additional forms of process data such as eye fixations (Chauliac et al, 2020), trace data (van Halem et al, 2020), and response times (Iaconelli & Wolters, 2020).…”