Fine-Grained Classroom Activity Detection from Audio with Neural Networks

Abstract

Instructors are increasingly incorporating student-centered learning techniques in their classrooms to improve learning outcomes. In addition to lecture, these class sessions involve forms of individual and group work, and greater rates of student-instructor interaction. Quantifying classroom activity is a key element of accelerating the evaluation and refinement of innovative teaching practices, but manual annotation does not scale. In this manuscript, we present advances to the young application area of automatic classroom activity detection from audio. Using a university classroom corpus with nine activity labels (e.g., "lecture," "group work," "student question"), we propose and evaluate deep fully connected, convolutional, and recurrent neural network architectures, comparing the performance of mel-filterbank, OpenSmile, and self-supervised acoustic features. We compare 9-way classification performance with 5-way and 4-way simplifications of the task and assess two types of generalization: (1) new class sessions from previously seen instructors, and (2) previously unseen instructors. We obtain strong results on the new fine-grained task and state-of-the-art on the 4-way task: our best model obtains frame-level error rates of 6.2%, 7.7% and 28.0% when generalizing to unseen instructors for the 4-way, 5-way, and 9-way classification tasks, respectively (relative reductions of 35.4%, 48.3% and 21.6% over a strong baseline). When estimating the aggregate time spent on classroom activities, our average root mean squared error is 1.64 minutes per class session, a 54.9% relative reduction over the baseline.

Speech analysis of teaching assistant interventions in small group collaborative problem solving with undergraduate engineering students

“…Other research has focused on distinguishing teacher and student speaker roles (Li et al, 2019), detecting common classroom activities (Donnelly et al, 2017), classifying student arguments (Lugini & Litman, 2019) or analysing dialogic instruction (Xu et al, 2020). Some studies have developed fine-grained classroom activity detection methods using audio data from webcams placed in instructors' workstations (Slyman et al, 2021).…”

Recognition of Suspicious Human Activity in Video Surveillance: A Review

“…The proposed work was tested on real environment using Siamese neural network for classification from classroom recordings. Authors in [39] also worked to improve the classroom learning outcome. A brief comparative summary of the relevant works is shown in Table I.…”