Multimodal Learning for Classroom Activity Detection

Abstract

Classroom activity detection (CAD) focuses on accurately classifying whether the teacher or student is speaking and recording both the length of individual utterances during a class. A CAD solution helps teachers get instant feedback on their pedagogical instructions. This greatly improves educators' teaching skills and hence leads to students' achievement. However, CAD is very challenging because (1) the CAD model needs to be generalized well enough for different teachers and students; (2) data from both vocal and language modalities has to be wisely fused so that they can be complementary; and (3) the solution shouldn't heavily rely on additional recording device. In this paper, we address the above challenges by using a novel attention based neural framework. Our framework not only extracts both speech and language information, but utilizes attention mechanism to capture long-term semantic dependence. Our framework is device-free and is able to take any classroom recording as input. The proposed CAD learning framework is evaluated in two real-world education applications. The experimental results demonstrate the benefits of our approach on learning attention based neural network from classroom data with different modalities, and show our approach is able to outperform state-of-the-art baselines in terms of various evaluation metrics.

What Classroom Audio Tells About Teaching: A Cost-effective Approach for Detection of Teaching Practices Using Spectral Audio Features

“…In educational research regarding teaching observation, the procedure is usually accompanied with audio or video recordings. However, a large portion of the work in this matter focuses on using linguistic features extracted from Automatic Speech Recognition (ASR) systems and Natural Language Processing (NLP) approaches for modelling teaching practices [19]. This causes these analyses to disregard important information besides the linguistic sources, such as paralinguistic or contextual ones, which are inferable from audio recordings [33], [10].…”

Automatic Detection of Gaze and Body Orientation in Elementary School Classrooms

“…One of the main problems is the slowness of the analysis. To encode using any of the established protocols, it is estimated that 4 h of coding are required for each hour of recording ( Li et al, 2020 ). Another difficulty is the dependence on the encoder.…”

Predicting Group Work Performance from Physical Handwriting Features in a Smart English Classroom

“…Using technology in classroom has shown to contribute to the development of students' creativity, motivation, and critical thought, encouraging also their capability of solving problems in a more collaborative way [30]. The increasing digital research in today's classrooms has encouraged a recent development of specific computer-based approaches for their application in E-learning environments, such as classroom activity detection [21], hand-rising gesture recognition [23], and classroom motion tracking [15]. Particularly, the proliferation of sensors in classrooms has created an environment in which students' behaviours are continuously monitored and recorded [2,28].…”