Granularity matters: comparing different ways of measuring self-regulated learning

Abstract

Although self-regulated learning (SRL) is becoming increasingly important in modern educational contexts, disagreements exist regarding its measurement. One particularly important issue is whether self-reports represent valid ways to measure this process. Several researchers have advocated the use of behavioral indicators of SRL instead. An outstanding research debate concerns the extent to which it is possible to compare behavioral measures of SRL to traditional ways of measuring SRL using self-report questionnaire data, and which of these methods provides the most valid and reliable indicator of SRL. The current review investigates this question. It was found that granularity is an important concept in the comparison of SRL measurements, influencing the degree to which students can accurately report on their use of SRL strategies. The results show that self-report questionnaires may give a relatively accurate insight into students' global level of self-regulation, giving them their own value in educational research and remediation. In contrast, when students are asked to report on specific SRL strategies, behavioral measures give a more accurate account. First and foremost, researchers and practitioners must have a clear idea about their research question or problem statement, before choosing or combining either form of measurement.

When Can Making a Drawing Hinder Problem Solving? Effect of the Drawing Strategy on Linear Overgeneralizations and Problem Solving

“…Thus, we did not have specific expectations. The aim of analyzing students' perceived performance is to increase the validity by using different indications of students' success (Rovers et al, 2019) and to gather further information that helps to explain the findings from our experimental study. Students' perceived performance in the drawing and visual monitoring conditions will provide indications of students' awareness of the non-linear property of the problems.…”

A mixed method approach to studying self-regulated learning in MOOCs

“…However, the interview results show that interpretation of trace data is often ambiguous. More research of the kind presented here should be conducted to establish what components of trace data can be validly labelled, and how they should be labelled, and what components of trace data will remain ambiguous (Rovers et al, 2019;Van Laer & Elen, 2018). To achieve this aim of enhancing the validity of SRL measurement, further research could make use not only of trace data and interview data as presented in the current study, but could incorporate promising data collection methods such as eye-tracking (Salmeron, Gil, & Bråten, 2018;Trevors, Feyzi-Behnagh, Azevedo, & Bouchet, 2016).…”

“…However, the same studies also demonstrate the challenges of using trace data as a measurement of SRL. The most striking problem is that while trace data shed light on leaners' behaviour in the online learning environment, there is doubt on how to interpret behaviour in terms of SRL (Cicchinelli et al, 2018;Jovanović et al, 2017;Min & Jingyan, 2017;Phillips et al, 2011;Rovers et al, 2019). SRL activities are for a large part covert in nature; they constitute the regulating activities that shape and guide the observable learning activities (Nelson & Narens, 1990).…”

“…A second challenge in measuring SRL with trace data is that some SRL activities take place outside of the online learning environment (Min & Jingyan, 2017). Winne and Jamieson-Noel (2002) for instance argued that scrolling of learners through a text-document indicated planning, planning may however also have been (mostly) a mental activity (Rovers et al, 2019). Veletsianos, Reich, and Pasquini (2016) studied learners' activities outside of the online learning environment and identified additional activities in three domains.…”

“…These so-called traces of learner behaviour are stored as time stamped events, providing an overview of all learners activities within the online learning environment. Using trace data as an indicator for learners' SRL is considered a promising approach because the data are gathered automatically, unobtrusively, and over longer durations of time (Rovers, Clarebout, Savelberg, De Bruin, & Van Merriënboer, 2019;Van Laer & Elen, 2018). Over the past years, the call for fine-grained process measures of SRL has strengthened (e.g., Winne, 2010) and researchers have therefore started using trace data to measure SRL (e.g., Cicchinelli et al, 2018;Van Laer & Elen, 2018).…”