Beyond First Encounters with Analytics: Questions, Techniques and Challenges in Instructors’ Sensemaking

Abstract

Despite growing implementation of teacher-facing analytics in higher education, relatively little is known about the detailed processes through which instructors make sense of analytics in their teaching practices beyond their initial encounters with tools. This study unpacked the sensemaking process of thirteen instructors with analytic experience, using interviews that included walkthroughs of their analytics use. Qualitative inductive analysis was used to identify themes related to (1) the questions they asked of the analytics, (2) the techniques they used to interpret them, and (3) the challenges they encountered. Findings indicated that instructors went beyond a general curiosity to develop three types of questions of the analytics (goal-oriented, problem-oriented, and instruction modification questions). Instructors also used specific techniques to read and explain data by (a) developing expectations about the answers the analytics would provide, and (b) making comparisons to reveal student diversity, identify effects of instructional revision and diagnose issues. The study found instructors faced an initial learning curve when seeking and making use of relevant information, but also continued to revisit these challenges when they were not able to develop a routine of analytics use. These findings both contribute to a conceptual understanding of instructor analytic sensemaking and have practical implications for its systematic support.

The equalizing effect of teacher dashboards on feedback in K-12 classrooms

“…As the interaction between a tool and the incorporation of it in daily routines is of great importance for the successful usage of new tools (Norman, 1991;Rienties et al, 2018), teachers who have more experience with dashboards could use the information differently. In a similar vein, Wise and Jung (2019) state "integrating the use of learning analytics into teaching practices to inform instructional decisions takes time" (p. 53) and teachers need to both understand the additional value of such dashboards and integrate them into their daily routines to achieve their full benefit (Li et al, 2021). Further research should therefore investigate how differences in teachers' experience influences their feedback practices.…”

The effects of visualisation literacy and data storytelling dashboards on teachers’ cognitive load

“…Moreover, identifying the types of questions teachers may answer based on the data displayed in a dashboard is essential for the effective use of a given LA dashboard (Li et al, 2021;Verbert et al, 2013). For instance, Li et al highlighted that various categories of questions from teachers can be addressed by different dashboard designs.…”

“…Furthermore, aside from the complexities of specific visualisations, the overall design of the dashboard interface, in terms of its complexity and intuitiveness, can also pose challenges to teachers, particularly those without a strong foundation in data analysis (Fernandez Nieto et al, 2022). Finally, it may be challenging for teachers to identify the types of educational questions they may want to answer through LA dashboard use (Li et al, 2021). Indeed, to fully benefit from LA dashboards, teachers may need a certain level of visualisation literacy (VL) (Pozdniakov et al, 2023).…”

Networks in Learning Analytics: Where Theory, Methodology, and Practice Intersect

“…Besides human networks, networks of words are also constructed based on their co-occurrence in online posts to prompt student reflection on collaborative knowledge building (Feng et al, 2021). While these interventions contribute novel toolkits for improving learning, emergent work in learning analytics has begun to shed light on the nuanced process of instructor sensemaking and pedagogical action-taking (Li et al, 2021;van Leeuwen et al, 2017), which are also important for network-based learning analytics projects.…”