Where is the Learning in Learning Analytics? A Systematic Literature Review on the Operationalization of Learning-Related Constructs in the Evaluation of Learning Analytics Interventions

Abstract

Learning technologies enable interventions in the learning process aiming to improve learning. Learning analytics provides such interventions based on analysis of learner data, which are believed to have beneficial effects on both learning and the learning environment. Literature reporting on the effects of learning analytics interventions on learning allows us to assess in what way learning analytics improves learning. No standard set of operational definitions for learning affected by learning analytics interventions is available. We performed a systematic literature review of 1932 search hits, which yielded 62 key studies. We analyzed how affected learning was operationalized in these key studies and classified operational definitions into three categories: 1) learning environment; 2) learning process; and 3) learning outcome. A deepening analysis yielded a refined classification scheme with 11 subcategories. Most of the analyzed studies relate to either learning outcome or learning process. Only nine of the key studies relate to more than one category. Given the complex nature of applying learning analytics interventions in practice, measuring the effects on a wider spectrum of aspects can give more insight into the workings of learning analytics interventions on the different actors, processes, and outcomes involved. Based on the results of our review, we recommend making deliberate decisions on the (multiple) aspects of learning one tries to improve by applying learning analytics. Our refined classification with examples of operational definitions may help both academics and practitioners doing so, as it allows for a more structured, grounded, and comparable positioning of learning analytics benefits.

The Challenge of Information and Communications Technology in Education

“…For example, testing on mobile devices is a similar issue since, without on-site supervision, teachers have little reason to trust that the student will participate honestly. Also, learning outcomes, such as Constructivist self-regulated learning skills, are challenging to measure, so learning analytics should move away from mere performance-based evaluation of learning analytics projects and include measurements related to learning processes and learning environment [47]. The effects of learning analytics on learning must be defined [47] as often the measured learning proxies, such as grades and LMS activity, can be overly narrow and miss much like student activity outside of school [48].…”

“…Also, learning outcomes, such as Constructivist self-regulated learning skills, are challenging to measure, so learning analytics should move away from mere performance-based evaluation of learning analytics projects and include measurements related to learning processes and learning environment [47]. The effects of learning analytics on learning must be defined [47] as often the measured learning proxies, such as grades and LMS activity, can be overly narrow and miss much like student activity outside of school [48]. Instead, an approach including assessment from multiple perspectives, such as a peers, self, teacher, parents, invited experts, and machine feedback can help provide a clearer picture of student learning.…”

A Deep Learning Model to predict the Industry Readiness of Engineering Students Community

“…A systematic literature1 [11] review about the learning strategies, evaluations and implementation beds were discussed. They had organized the literature through three different sections.…”

“…Also they have identified the behavioral analytics of the interdisciplinary users of the system. Justian and Esther [11] listed the systematic survey of learning teachnologies in terms of learning environments, learning process and learning outcome. Bart et al [12] created a learning frameworks which evaluates the students community using design and evidence based research.…”

2016-2020 yılları arasında Öğrenme Analitiği ile ilgili Yapılmış SSCI İndeksli Makalelerin Sistematik Olarak İncelenmesi

“…Öğrenme analitiği geleneksel eğitim araştırmalarından farklı olarak, mevcut öğrenme uygulamalarını ve öğrencileri bilgilendirirken genelleştirilebilir bilgiyi de üretir ve daha sonra bunu gelecekteki öğrencilerin deneyimlerini geliştirmek için kullanır (Clow 2012;Ochoa & Wise, 2020). Öğrenme analitikleri sürecinde elde edilen verilerin öğrenme ve öğrenme ortamı üzerinde yararlı etkileri olmaktadır (Knobbout & Stappen, 2020).…”