Student Behavior Recognition System for the Classroom Environment Based on Skeleton Pose Estimation and Person Detection

Abstract

Human action recognition has attracted considerable research attention in the field of computer vision, especially for classroom environments. However, most relevant studies have focused on one specific behavior of students. Therefore, this paper proposes a student behavior recognition system based on skeleton pose estimation and person detection. First, consecutive frames captured with a classroom camera were used as the input images of the proposed system. Then, skeleton data were collected using the OpenPose framework. An error correction scheme was proposed based on the pose estimation and person detection techniques to decrease incorrect connections in the skeleton data. The preprocessed skeleton data were subsequently used to eliminate several joints that had a weak effect on behavior classification. Second, feature extraction was performed to generate feature vectors that represent human postures. The adopted features included normalized joint locations, joint distances, and bone angles. Finally, behavior classification was conducted to recognize student behaviors. A deep neural network was constructed to classify actions, and the proposed system was able to identify the number of students in a classroom. Moreover, a system prototype was implemented to verify the feasibility of the proposed system. The experimental results indicated that the proposed scheme outperformed the skeleton-based scheme in complex situations. The proposed system had a 15.15% higher average precision and 12.15% higher average recall than the skeleton-based scheme did.

Design and Exploration of College English Reading, Writing, and Translation Teaching Classroom Based on Machine Learning

“…where x max ðy max Þ represents the maximum xðyÞ value extracted from all joint coordinates of each individual, and x min ðy min Þ is the minimum xðyÞ value. Sometimes, the coordinate of the key joint points cannot be extracted when local key points are unidentifiable because of specific data, and the results are output as 0; such data need to be deleted from the extraction process since when the coordinates of this key point are unidentifiable, the minimum of the normalization equation gets 0, and the data normalization will be deviated, which will have a great negative impact on subsequent training and learning [21]. In the follow-up, the normalized data will be labeled and stored in the corresponding files to serve as the data training set for the ML algorithm.…”

Student Dangerous Behavior Detection in School

“…Both of object detection and pose estimation have been used for action recognition independently, but rarely together. Several works with similar opinions to ours were proposed, such as UAV surveillance system [4], Drone Surveillance System (DSS) [5] and student behavior recognition system [6], which cannot be trained end-to-end.…”

“…Based on blocks C 3 ∼ C 5 , we construct the feature pyramid P 3 ∼ P 7 following FPN [7]. We share the heads between different feature levels of P i (i ∈ [3,4,5,6,7]) for multi-level prediction. To regress different size range among P i , we increase a trainable scalar s i to automatically adjust the base of the exponential function exp(s i x) for feature level P i .…”

Classroom Behavior Detection Based on Improved YOLOv5 Algorithm Combining Multi-Scale Feature Fusion and Attention Mechanism

“…The authors of [25] proposed a student body gesture recognition method based on the Fisher Broad learning system, and defined seven learning behaviors, which achieved good results on the selfbuilt dataset. The authors of [26] used OpenPose framework to collect students' skeleton information, build a neural network to classify the extracted skeletal data by normalizing joint position, joint distance, and skeletal angle, and propose a student behavior recognition system based on human skeleton estimation and person detection. The authors of [27] built a deep convolutional neural network to identify head poses, used cascaded facial feature point positioning to extract facial expression key points, and distinguished students' classroom behaviors by combining head poses and facial expressions.…”