Adapting MobileNets for mobile based upper body pose estimation

Abstract

Human pose estimation through deep learning has achieved very high accuracy over various difficult poses. However, these are computationally expensive and are often not suitable for mobile based systems. In this paper, we investigate the use of MobileNets, which is well-known to be a lightweight and efficient CNN architecture for mobile and embedded vision applications. We adapt MobileNets for pose estimation inspired by the hourglass network. We introduce a novel split stream architecture at the final two layers of the MobileNets. This approach reduces over-fitting, resulting in improvement in accuracy and reduction in parameter size. We also show that by maintaining part of the original network we are able to improve accuracy by transferring the learned features from ImageNet pre-trained MobileNets. The adapted model is evaluated on the FLIC dataset. Our network out-performed the default MobileNets for pose estimation, as well as achieved performance comparable to the state of the art results while reducing inference time significantly.

Monocular human pose estimation: A survey of deep learning-based methods

“…This network is similar to the idea of the Hourglass network while utilizing U-Net as each component with a more optimized global connection across each stage resulting in fewer parameters and small model size. Debnath et al (2018) adapted MobileNets (Howard et al, 2017) for pose estimation by designing a split stream architecture at the final two layers of the MobileNets. Feng et al (2019) designed a lightweight variant of Hourglass network and trained it with a full teacher Hourglass network by a Fast Pose Distillation (FPD) training strategy.…”

“…(1) Patch-based: (Jain et al, 2013;Chen and Yuille, 2014;Ramakrishna et al, 2014) (2) Network design: (Tompson et al, 2015;Bulat and Tzimiropoulos, 2016;Xiao et al, 2018), multi-scale inputs (Rafi et al, 2016), heatmap-based improvement (Papandreou et al, 2017), Hourglass (Newell et al, 2016), CPM (Wei et al, 2016), PRM (Yang et al, 2017), feed forward module (Belagiannis and Zisserman, 2017), HRNet (Sun et al, 2019), GAN (Chou et al, 2017;Peng et al, 2018) (3) Body structure constraint: (Tompson et al, 2014;Lifshitz et al, 2016;Yang et al, 2016;Gkioxari et al, 2016;Chu et al, 2016Chu et al, , 2017Ning et al, 2018;Ke et al, 2018;Tang et al, 2018a;Tang and Wu, 2019) (4) Temporal constraint: (Jain et al, 2014;Pfister et al, 2015;Luo et al, 2018) (5) Network compression: (Tang et al, 2018b;Debnath et al, 2018;Feng et al, 2019) 2D Multiple…”

Human Body Pose Estimation for Gait Identification: A Comprehensive Survey of Datasets and Models

“…The SPPE regression methods currently perform lower as compared to the body part detection methods. For example, the best accuracy was reported by Debnath et al [34] with a PCKh@0.2 of 96.4%. Although body part detection methods have shown excellent performance, however, they are prone to estimating false positives [143].…”

“…An alternative approach is to use heatmaps which provide richer supervision information compared to joint coordinates, by preserving spatial location information [220]. This information is ideal for training CNNs and has resulted in a growing interest in leveraging CNNs for the purpose of HPE [5,13,17,34,45,89,97,124,146,178,201,206,209]. Table 6 shows that the best performing body part detection-based method is achieved by Debnath et al [34] with a PCKh@0.2 of 96.4%.…”

Alignment control using visual servoing and mobilenet single-shot multi-box detection (SSD): a review

“…MobileNets reduce computation in the first few layers by embracing depthwise separable convolutions and inception models. The embedded pointwise convolution factorizes standard convolution into a 1×1 convolution and depth-wise convolution, which reduces computation and model size [30]. Therefore, MobileNets institute autonomous behavior into systems to reduce execution and cognitive burden on users by facilitating remote inspection and package delivery, besides effectively surveying hostile environments.…”