No abstract

Multimodal Learning Analytics (MMLA) researchers are progressively employing machine learning (ML) techniques to develop predictive models to improve learning and teaching practices. These predictive models are often evaluated for their generalizability using methods from the ML domain, which do not take into account MMLA’s educational nature. Furthermore, there is a lack of systematization in model evaluation in MMLA, which is also reflected in the heterogeneous reporting of the evaluation results. To overcome these issues, this paper proposes an evaluation framework to assess and report the generalizability of ML models in MMLA (EFAR-MMLA). To illustrate the usefulness of EFAR-MMLA, we present a case study with two datasets, each with audio and log data collected from a classroom during a collaborative learning session. In this case study, regression models are developed for collaboration quality and its sub-dimensions, and their generalizability is evaluated and reported. The framework helped us to systematically detect and report that the models achieved better performance when evaluated using hold-out or cross-validation but quickly degraded when evaluated across different student groups and learning contexts. The framework helps to open up a “wicked problem” in MMLA research that remains fuzzy (i.e., the generalizability of ML models), which is critical to both accumulating knowledge in the research community and demonstrating the practical relevance of these techniques.

Monitoring and guiding multiple groups of students in face-to-face collaborative work is a demanding task which could possibly be alleviated with the use of a technological assistant in the form of learning analytics. However, it is still unclear whether teachers would indeed trust, understand, and use such analytics in their classroom practice and how they would interact with such an assistant. The present research aimed to find out what the perception of in-service secondary school teachers is when provided with a dashboard based on audio and digital trace data when monitoring a collaborative learning activity. In a vignette study, we presented twenty-one in-service teachers with videos from an authentic collaborative activity, together with visualizations of simple collaboration analytics of those activities. The teachers perceived the dashboards as providers of useful information for their everyday work. In addition to assisting in monitoring collaboration, the involved teachers imagined using it for picking out students in need, getting information about the individual contribution of each collaborator, or even as a basis for assessment. Our results highlight the need for guiding dashboards as only providing new information to teachers did not compel them to intervene and additionally, a guiding dashboard could possibly help less experienced teachers with data-informed assessment.

No abstract

The estimation of collaboration quality using manual observation and coding is a tedious and difficult task. Researchers have proposed the automation of this process by estimation into few categories (e.g., high vs. low collaboration). However, such categorical estimation lacks in depth and actionability, which can be critical for practitioners. We present a case study that evaluates the feasibility of quantifying collaboration quality and its multiple sub-dimensions (e.g., collaboration flow) in an authentic classroom setting. We collected multimodal data (audio and logs) from two groups collaborating face-to-face and in a collaborative writing task. The paper describes our exploration of different machine learning models and compares their performance with that of human coders, in the task of estimating collaboration quality along a continuum. Our results show that it is feasible to quantitatively estimate collaboration quality and its sub-dimensions, even from simple features of audio and log data, using machine learning. These findings open possibilities for in-depth automated quantification of collaboration quality, and the use of more advanced features and algorithms to get their performance closer to that of human coders.