Pose Mask: A Model-Based Augmentation Method for 2D Pose Estimation in Classroom Scenes Using Surveillance Images

Abstract

Solid developments have been seen in deep-learning-based pose estimation, but few works have explored performance in dense crowds, such as a classroom scene; furthermore, no specific knowledge is considered in the design of image augmentation for pose estimation. A masked autoencoder was shown to have a non-negligible capability in image reconstruction, where the masking mechanism that randomly drops patches forces the model to build unknown pixels from known pixels. Inspired by this self-supervised learning method, where the restoration of the feature loss induced by the mask is consistent with tackling the occlusion problem in classroom scenarios, we discovered that the transfer performance of the pre-trained weights could be used as a model-based augmentation to overcome the intractable occlusion in classroom pose estimation. In this study, we proposed a top-down pose estimation method that utilized the natural reconstruction capability of missing information of the MAE as an effective occluded image augmentation in a pose estimation task. The difference with the original MAE was that instead of using a 75% random mask ratio, we regarded the keypoint distribution probabilistic heatmap as a reference for masking, which we named Pose Mask. To test the performance of our method in heavily occluded classroom scenes, we collected a new dataset for pose estimation in classroom scenes named Class Pose and conducted many experiments, the results of which showed promising performance.

Masked Autoencoders in Computer Vision: A Comprehensive Survey

“…Marlin et al [97] (2023.6) also employ MAE for learning facial features, with a focus on facial details and facial features. PoseMask [98] (2022.10) applies MAE for pose estimation in classroom scenarios, using heatmaps as reference masks to estimate poses in crowded or occluded scenes. Furthermore, Sheng et al [99] (2022.10) use MAE for feature extraction of gestures in Spatial-Temporal Motion Maps (STMM), improving gesture recognition accuracy.…”

“…This means that even if the reconstructed results differ from the original, they still possess a certain level of coherence and can connect with contextual information. Therefore, MAE has significant applications in image generation [98], [104], [110], and also performs well in tasks that require temporal coherence [4], [135].…”