Toward Automated Classroom Observation: Predicting Positive and Negative Climate

Abstract

In this work we present a multi-modal machine learning-based system, which we call ACORN, to analyze videos of school classrooms for the Positive Climate (PC) and Negative Climate (NC) dimensions of the CLASS [1] observation protocol that is widely used in educational research. ACORN uses convolutional neural networks to analyze spectral audio features, the faces of teachers and students, and the pixels of each image frame, and then integrates this information over time using Temporal Convolutional Networks. The audiovisual ACORN's PC and NC predictions have Pearson correlations of 0.55 and 0.63 with ground-truth scores provided by expert CLASS coders on the UVA Toddler dataset (cross-validation on n = 300 15-min video segments), and a purely auditory ACORN predicts PC and NC with correlations of 0.36 and 0.41 on the MET dataset (test set of n = 2000 videos segments). These numbers are similar to inter-coder reliability of human coders. Finally, using Graph Convolutional Networks we make early strides (AUC=0.70) toward predicting the specific moments (45-90sec clips) when the PC is particularly weak/strong. Our findings inform the design of automatic classroom observation and also more general video activity recognition and summary recognition systems.

Multimodal Learning Analytics research with young children: A systematic review

“…The availability and minimal intrusiveness of video cameras (compact cameras that may be smaller than a coin or incorporated in any mobile device), in conjunction with the potential of machine learning, makes them a device that is commonly used for studying aspects of understanding, achievement, emotions, attention and speech in MMLA research with children. Children’s actions can be recorded in a wide‐angle shot (Pérez, Martínez, Avila & Espinosa, 2018; Ramakrishnan et al ., 2019), with a 360° point‐of‐view multi‐camera system (Malmberg et al ., 2019) or with a webcam on top of a computer screen (Chen et al ., 2016; Pereira et al ., 2018; Sharma et al ., 2019) to analyse the emotions generated during the learning process automatically. Spikol and colleagues (2018) calculated the distance between the participants (recorded with a front camera), which they treated as a proxy for collaboration.…”

“…Rudovic, Lee, Dai, Schuller and Picard, (2018) used cameras in a robot that interacted with children to conduct real‐time extrapolation and processing of the LA from facial expressions, head movements, body movements, posture and gestures. Although facial recognition is widely used in the reviewed literature (Andrade & Worsley, 2017; Chen et al ., 2016; Ouherrou et al ., 2019; Pereira et al ., 2018; Ramakrishnan et al , 2019; Sharma et al ., 2019; Spikol et al ., 2018), Ouherrou and colleagues (2019) highlight the constraints of using it alone compared to several integrated systems (such as head movement or tone of voice), which could be more effective.…”

“…Algorithms and statistical models allow video and audio recordings to be analysed, and as far as hardware is concerned only a camera system is needed. This is possibly the most anticipated short‐term breakthrough for MMLA research with children, but according to Ramakrishnan and colleagues “two inter‐related challenges posed by this task are data privacy and data scarcity” (2019, p. 2). Some technology, such as eye‐tracking devices, seems to be simply incompatible with the study of young children learning in natural contexts. The use of these devices is intended for adults, that is, for conscious and critical research participants with a capacity for self‐control that children under 6 years old do not possess.…”

“…Indeed, the multiple options and combinations of resources offered by MMLA allows the classroom observation protocol, which is manually filled in by the same kindergarten teachers, to be complemented with automatic data from MMLA. These observation protocols “are invaluable for providing feedback to teachers on what went well in their teaching and what could be improved” (Ramakrishnan, Ottmar, LoCasale‐Crouch & Whitehill, 2019, p. 1) and the MMLA would reduce their cost and time, also increasing their reliability. At the same time, it would allow the teacher to focus his assessment on more specific aspects of executive functions such as selective attention span, working memory, etc.…”

Measuring Equity-Promoting Behaviors in Digital Teaching Simulations: A Topic Modeling Approach

“…With NLP, researchers are able to quantitatively analyze text data for patterns that are predictive of participants’ attitudes, behaviors, and practices. NLP tools have been used, for example, to analyze transcripts of classroom teaching to detect the presence of particular instructional and discourse moves or to evaluate classroom climate (Jensen et al, 2020; Ramakrishnan et al, 2019).…”