Building the foundations for measuring learning gain in higher education: a conceptual framework and measurement instrument

Abstract

In this paper, we set out the first step towards the measurement of learning gain in higher education by putting forward a conceptual framework for understanding learning gain that is relevant across disciplines. We then introduce the operationalisation of this conceptual framework into a new set of measurement tools. With the use of data from a large-scale survey of 11 English universities and over 4,500 students, we test the reliability and validity of the measurement instrument empirically. We find support in the data for the reliability of most of the measurement scales we put forward, as well as for the validity of the conceptual framework. Based on these results, we reflect on the conceptual framework and associated measurement tools in the context of at-scale deployment and the potential implications for policy and practice in higher education. ARTICLE HISTORY

Synergies of Learning Analytics and Learning Design: A Systematic Review of Student Outcomes

“…It is well aligned with current issues and common goals in the HE sector globally, which increasingly calls for the use of data to improve student outcomes in employability, equitable access, lifelong learning, and digital literacy. The "Learning Gains in HE" model (Vermunt, Ilie, & Vignoles, 2018) provides robust conceptions of cognitive, metacognitive, socio-communicative, and affective learning that have been validated across students' subjectspecific and non-subject-specific knowledge, skills, and attitudes. This classification aided the mapping of LD effects on students' cognitive and behavioural learning processes reported in the primary studies, answering a call for the inclusion of richer LA data (e.g., Tempelaar, Rienties, Mittelmeier, & Nguyen, 2018) to inform more nuanced teacher actions.…”

“…The selected studies were categorized in two ways: first, thematically according to three perspectives adapted from Redecker et al (2011; see Supplementary Figure S1 for the original framework), and, second, according to effects on learning across four components in light of Vermunt et al's (2018) framework of learning gains in HE.…”

“…However, Cohen's d is commonly used for comparison purposes in meta-analyses and can guide the reader toward the main messages contained in the data. Extracted effect sizes were categorized qualitatively into four perspectives of learning gains (Vermunt et al, 2018), which may add a level of subjectivity. Furthermore, the inconsistency and lack of an overarching definition for student learning outcome resulted in a broad categorization of effects on learning performance, course outcome, and online presence, which may be invalid as a general definition of outcome.…”

“…Furthermore, the inconsistency and lack of an overarching definition for student learning outcome resulted in a broad categorization of effects on learning performance, course outcome, and online presence, which may be invalid as a general definition of outcome. The intention was to aid the interpretation of results based on pragmatic themes according to existing frameworks (Redecker et al, 2011;Vermunt et al, 2018) and to ensure external validity. Another limitation is that qualitative studies that investigated the effects of analytics-informed LDs on students' perceived performance and learning were not included.…”

Where is the Learning in Learning Analytics? A Systematic Literature Review on the Operationalization of Learning-Related Constructs in the Evaluation of Learning Analytics Interventions

“…The concept of learning gain captures the idea that learning is visible through a change over time in a learner's behavior, attitude, and/or knowledge. There is no standard definition, conceptualization, or measurement (instrument) to assess learning gain; a conceptual framework with a set of measurement tools is currently being developed for English higher educational institutes [96]. In the conceptual framework proposed in [96], a distinction is made between four components (cognitive, metacognitive, affective, and socio-communicative) and three cross-cutting dimensions (view of knowledge and learning, research attitude, and moral reasoning).…”