Do Teachers Find Dashboards Trustworthy, Actionable and Useful? A Vignette Study Using a Logs and Audio Dashboard

Abstract

Monitoring and guiding multiple groups of students in face-to-face collaborative work is a demanding task which could possibly be alleviated with the use of a technological assistant in the form of learning analytics. However, it is still unclear whether teachers would indeed trust, understand, and use such analytics in their classroom practice and how they would interact with such an assistant. The present research aimed to find out what the perception of in-service secondary school teachers is when provided with a dashboard based on audio and digital trace data when monitoring a collaborative learning activity. In a vignette study, we presented twenty-one in-service teachers with videos from an authentic collaborative activity, together with visualizations of simple collaboration analytics of those activities. The teachers perceived the dashboards as providers of useful information for their everyday work. In addition to assisting in monitoring collaboration, the involved teachers imagined using it for picking out students in need, getting information about the individual contribution of each collaborator, or even as a basis for assessment. Our results highlight the need for guiding dashboards as only providing new information to teachers did not compel them to intervene and additionally, a guiding dashboard could possibly help less experienced teachers with data-informed assessment.

Scalability, Sustainability, and Ethicality of Multimodal Learning Analytics

“…In education, this means that support tools that are sustainable can be used throughout multiple school years and can adapt, to some extent, to changes in learning designs [35]. The sustainability of MMLA systems can be investigated by measuring the extent to which the technology has reached a readiness level [62] that can allow educational stakeholders to use the technology without the support from researchers and technology developers [61]; and the extent to which MMLA systems can be adapted according to the changes in learning designs while maintaining their functionality [32]. Unsustainable educational technologies could pose additional resource burdens on institutions, for example, on human resources (e.g., technicians and data scientists) required to support the ongoing operation of the MMLA innovations [61].…”

Teacher Artificial Intelligence-Supported Pedagogical Actions in Collaborative Learning Coregulation: A Wizard-of-Oz Study

“…The boxplot in Figure 3 shows that the overall knowledge has increased. A paired t-test did not show statistical significance, but as our previous study (Kasepalu et al, 2021) showed that experience was a critical factor in the projected use of a tool, we wanted to see if there was a difference in the results based on the level of experience. Using the experience threshold from The Teaching and Learning International Survey (TALIS, 2018), there were eight teachers with less than 5 years of experience and nine teachers with more than 5 years of teaching experience, three teachers had not answered the pre-and post-test.…”

“…We have completed a needs analysis and paper prototype in the first iteration (Kasepalu et al, 2019), a research prototype multimodal learning analytics (MMLA) tool named CoTrack in the second iteration (Chejara et al, 2020). The third iteration, a vignette study with twenty-one inservice teachers, showed a discrepancy between the evaluation of the quality of collaboration among teachers, i.e., interviewed teachers were unable to agree whether the perceived collaboration had been effective or not (Kasepalu et al, 2021). With the purpose of assessing the quality of the collaboration process, Rummel et al (2011) have proposed the Adaptable Collaboration Analysis Rating Scheme (ACARS).…”

“…In the first iteration of our DBR process we looked into the problems teachers had with CL (see Kasepalu et al, 2021 for more details). After this, we mapped these problems with the seven dimensions proposed by Rummel et al (2011) (see Appendix 1 for the whole model, Table 1 for an excerpt).…”

“…This paper is part of a larger design-based research (DBR), where empirical educational research is combined with the theory-driven design of learning environments (Baumgartner et al, 2003) with the purpose of finding solutions to support teachers during CL using LA. In prior work, we have completed three DBR iterations (Kasepalu et al, 2021), this being the fourth iteration process, where the utility of an AI assistant was explored using a paper prototype. The purpose of this study was to investigate the perceived usefulness and pedagogical actions of teachers by conducting a Wizard-of-Oz (WoZ) protocol with twenty in-service teachers from thirteen different upper-secondary and vocational schools in Estonia.…”