Pose estimator and tracker using temporal flow maps for limbs

Hwang, Jee Youn; Lee, Jieun; Park, Sungheon; Kwak, Nojun

doi:10.1109/ijcnn.2019.8851734

Cited by 36 publications

(13 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, the feature representation should (iii) focus more on appearance rather than location, since objects can move in time. Normally, such a feature embedding can be learned, e.g., via a side branch trained with labelled identities across frames as the supervision signal, as is evidenced in several existing works [5,19,50]. Although no such annotations are accessible here, we find that instance-level annotation on images already provides sufficient information to achieve the above goals, i.e., to distinguish which pixels belong to the same instance, and which are from different ones.…”

Section: Instance Contrastive Lossmentioning

confidence: 84%

Learning to Track Instances without Video Annotations

Fu¹,

Liu²,

Iqbal³

et al. 2021

Preprint

View full text Add to dashboard Cite

Tracking segmentation masks of multiple instances has been intensively studied, but still faces two fundamental challenges: 1) the requirement of large-scale, frame-wise annotation, and 2) the complexity of two-stage approaches.To resolve these challenges, we introduce a novel semisupervised framework by learning instance tracking networks with only a labeled image dataset and unlabeled video sequences. With an instance contrastive objective, we learn an embedding to discriminate each instance from the others. We show that even when only trained with images, the learned feature representation is robust to instance appearance variations, and is thus able to track objects steadily across frames. We further enhance the tracking capability of the embedding by learning correspondence from unlabeled videos in a self-supervised manner. In addition, we integrate this module into single-stage instance segmentation and pose estimation frameworks, which significantly reduce the computational complexity of tracking compared to two-stage networks. We conduct experiments on the YouTube-VIS and PoseTrack datasets. Without any video annotation efforts, our proposed method can achieve comparable or even better performance than most fullysupervised methods 1 .

show abstract

Section: Instance Contrastive Lossmentioning

confidence: 84%

Learning to Track Instances without Video Annotations

Fu¹,

Liu²,

Iqbal³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The AP metric follows the same protocol of human instances, but to the best of our knowledge no previous method has evaluated AP on ApolloCar3D [58] without leveraging 3D information. Hence, we include a study PoseTrack 2018 MOTA FPS openSVAI [28] 54.5 -MIPAL [26] 54.9 -Miracle [30] 57.4 -MSRA/FlowTrack [19] 61.4 0.7 OpenPifPaf (ours) 61.7 12.2 (a)…”

Section: Resultsmentioning

confidence: 99%

“…PoseTrack 2017 MOTA FPS STAF [27] 53.8 3 MIPAL [26] 54.5 -MSRA/FlowTrack [19] 57.8 0.7 HRNet [20] 57.9 -LightTrack [29] 58.0 -OpenPifPaf (ours) 60.6 12.2 KeyTrack [32] 61.2 1.0 DetTrack (offline) [67] 64.1 -…”

Section: Resultsmentioning

confidence: 99%

“…The temporal flow fields (TFF) feed into an edge cost computation for bipartite graph matching for tracking. The idea is extended in MIPAL [26] for tracking limbs instead of joints and in STAF [27].…”

Section: B Pose Trackingmentioning

confidence: 99%

“…Our method is bottom-up in both pose estimation and tracking and estimates temporal and spatial connections within a single stage. Most existing work -even other bottom-up tracking methods [10], [26] -employ a two stage process where, first, spatial connections are estimated and, second, temporal connections are made.…”

Section: B Pose Trackingmentioning

confidence: 99%

See 2 more Smart Citations

OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association

Kreiss¹,

Bertoni²,

Alahi³

2021

Preprint

View full text Add to dashboard Cite

Many image-based perception tasks can be formulated as detecting, associating and tracking semantic keypoints, e.g., human body pose estimation and tracking. In this work, we present a general framework that jointly detects and forms spatio-temporal keypoint associations in a single stage, making this the first real-time pose detection and tracking algorithm. We present a generic neural network architecture that uses Composite Fields to detect and construct a spatio-temporal pose which is a single, connected graph whose nodes are the semantic keypoints (e.g., a person's body joints) in multiple frames. For the temporal associations, we introduce the Temporal Composite Association Field (TCAF) which requires an extended network architecture and training method beyond previous Composite Fields. Our experiments show competitive accuracy while being an order of magnitude faster on multiple publicly available datasets such as COCO, CrowdPose and the PoseTrack 2017 and 2018 datasets. We also show that our method generalizes to any class of semantic keypoints such as car and animal parts to provide a holistic perception framework that is well suited for urban mobility such as self-driving cars and delivery robots.

show abstract

Personalized motion kernel learning for human pose estimation

Wang

Feng

Chen

et al. 2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

Estimating human poses from a video is at the foundation of many visual intelligent systems. Various convolutional neural networks have been proposed, achieving state‐of‐the‐art performance on different image datasets. However, most existing approaches are image based, which deliver unreliable estimations on videos since they fail to model temporal consistency across video frames. Recently, another line of work leverages temporal cues for multi‐frame person pose estimation, yet still in an instance‐unaware fashion, disregarding the specific traits of different instances (persons) or different joints. In this paper, we propose a novel approach to learn specific keypoint motion representations for each person, termed Personalized Motion‐Aware Network (PMAN). In the PMAN, we devise three components: (i) an Instance‐Sensitive Extractor that adaptively computes the spatial features according to human physical characteristics; (ii) a Keypoint Motion Encoder that separately generates convolution kernels with fine‐grained keypoint motion encoding; (iii) a Motion Driven Decoder that parses multi‐frame spatial features of the same person to provide precise human pose estimations. Extensive experiments on PoseTrack2017 and PoseTrack2018 datasets demonstrate that our approach greatly improves the performance of multi‐frame human pose estimation. It is worth mentioning that our approach surpasses the state‐of‐the‐art method by +1.7 mAP and achieves 82.9 mAP on PoseTrack2017 dataset.

show abstract

Pose estimator and tracker using temporal flow maps for limbs

Cited by 36 publications

References 25 publications

Learning to Track Instances without Video Annotations

Learning to Track Instances without Video Annotations

OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association

Personalized motion kernel learning for human pose estimation

Contact Info

Product

Resources

About