Body-Part-Aware and Multitask-Aware Single-Image-Based Action Recognition

Abstract

Action recognition is an application that, ideally, requires real-time results. We focus on single-image-based action recognition instead of video-based because of improved speed and lower cost of computation. However, a single image contains limited information, which makes single-image-based action recognition a difficult problem. To get an accurate representation of action classes, we propose three feature-stream-based shallow sub-networks (image-based, attention-image-based, and part-image-based feature networks) on the deep pose estimation network in a multitasking manner. Moreover, we design the multitask-aware loss function, so that the proposed method can be adaptively trained with heterogeneous datasets where only human pose annotations or action labels are included (instead of both pose and action information), which makes it easier to apply the proposed approach to new data on behavioral analysis on intelligent systems. In our extensive experiments, we showed that these streams represent complementary information and, hence, the fused representation is robust in distinguishing diverse fine-grained action classes. Unlike other methods, the human pose information was trained using heterogeneous datasets in a multitasking manner; nevertheless, it achieved 91.91% mean average precision on the Stanford 40 Actions Dataset. Moreover, we demonstrated the proposed method can be flexibly applied to multi-labels action recognition problem on the V-COCO Dataset.

GaitVision: Real‐Time Extraction of Gait Parameters Using Residual Attention Network

“…[20] implemented simple image-based action recognition based on an HRNet [21] human pose estimation network. HRNet [21] represents multitasking features with a set of feature maps extracted from an image in the decreasing order of resolution and increasing order of channels [20].…”

“…Attention Image-Based Feature Stream. Bhandari et al [20] and Fukui et al [22] used attention mechanisms such as attention-based image feature extraction streams foreground analysis. is method uses an attention image-based feature stream [23], which is built on top of ResNet18 [24].…”

“…For accurate action recognition using each part, Bhandari et al [20] proposed a part-image-based feature stream.…”

Multi-Expert Human Action Recognition with Hierarchical Super-Class Learning

“…As reported in Table 6, SCLAR yields the best performance in terms of accuracy and efficiency in comparison with state-of-the-art works. More specifically, as a runner-up, SCLAR yields gains of 0.41% and 0.66% on mAP compared to the second [19] and third [54] best methods. While the obtained gains in respect to the second-best method [19] is not significantly pronounced, the efficiency gain is dramatically considerable.…”

“…Yet, [19] and [54] require additional supervision, such as human bounding boxes or predefined body parts, which confine their practical applications to specific domains with available bounding boxes. Besides, it is proven that SCLAR outperforms other studies [12,[20][21][22] to a great degree.…”