Facilitating the Use of Data From Multiple Sources for Formative Learning in the Context of Digital Assessments: Informing the Design and Development of Learning Analytic Dashboards

Abstract

Learning analytic dashboards (LADs) are data visualization systems that use dynamic data in digital learning environments to provide students, teachers, and administrators with a wealth of information about student’s engagement, experiences, and performance on tasks. LADs have become increasingly popular, particularly in formative learning contexts, and help teachers make data-informed decisions about a student’s developing skills on a topic. LADs afford the possibility for teachers to obtain real-time data on student performance, response processes, and progress on academic learning tasks. However, data presented on LADs are often not based on an evaluation of stakeholder needs, and have been found to not be clearly interpretable and actionable for teachers to readily adapt their pedagogical actions based on these insights. We elaborate on how insights from research focused on interpretation and use of Score Reporting systems and research on open learner models (OLMs) can be used to inform a research agenda aimed at exploring the design and evaluation of LADs.

Challenges and opportunities in score reporting: a panel of personal perspectives

“…Regarding the amount of data available using digital learning assessment and learning systems, the experts mentioned the opportunities that additional multimodal data will afford (e.g., assessing motivation and detecting cheating) and warned us about the need for developing appropriate methods that help us analyze rich process data to support decision making. Also, the authors elaborated on the importance of designing reporting systems (or dashboards) that take into account the needs of the audience and offer support for appropriate interpretation (Kannan and Zapata-Rivera, 2022).…”

“…There is already some interesting work in this area, and several Learning Analytic Dashboards (LADs) are being designed (Molenaar and Knoop-van Campen, 2018;Michaeli et al, 2020;Sahin and Ifenthaler, 2021) with the intent of providing real-time feedback on instructionally informative data such as students' time on task, progress toward goals, their overall level of conceptual understanding, and their strengths and needs relative to ongoing formative goals. This information can be scaffolded and presented to teachers in a real-time actionable dashboard (examples in Kannan et al, 2019;Kannan and Zapata-Rivera, 2022) with scaffolds and visualizations that can help teachers in tailoring their instruction to fill specific gaps in students' conceptual understandings.…”

“…SS: Interactive reports and dashboards aim to produce a wealth of information about test-taker's engagement, experiences, and performance on tasks (Kannan and Zapata-Rivera, 2022). Users of interactive reports and dashboards most likely would want to see the results on various aspects of learning and various types of interaction between a test-taker and the system, on the difference in performance, of the same group over multiple time periods, over multiple groups for the same time period, over multiple groups over several time periods, and often at the subscale level.…”