Learning to Detect and Track Visible and Occluded Body Joints in a Virtual World

Abstract

Multi-PeopleTracking in an open-world setting requires a special effort in precise detection. Moreover, temporal continuity in the detection phase gains more importance when scene cluttering introduces the challenging problems of occluded targets. For the purpose, we propose a deep network architecture that jointly extracts people body parts and associates them across short temporal spans. Our model explicitly deals with occluded body parts, by hallucinating plausible solutions of not visible joints. We propose a new end-to-end architecture composed by four branches (visible heatmaps, occluded heatmaps, part affinity fields and temporal affinity fields) fed by a time linker feature extractor. To overcome the lack of surveillance data with tracking, body part and occlusion annotations we created the vastest Computer Graphics dataset for people tracking in urban scenarios by exploiting a photorealistic videogame. It is up to now the vastest dataset (about 500.000 frames, almost 10 million body poses) of human body parts for people tracking in urban scenarios. Our architecture trained on virtual data exhibits good generalization capabilities also on public real tracking benchmarks, when image resolution and sharpness are high enough, producing reliable tracklets useful for further batch data association or re-id modules.

Human Pose Estimation for Real-World Crowded Scenarios

“…On the other hand, we observe that adding person instances or body parts is less effective and also requires additional attention to avoid creating erroneous poses in a top-down method. Our results further prove that JTA [3] is a valuable addition to the training data for crowd-level pose estimation, especially since no large datasets are available for pose estimation in crowded scenarios. The created extension of JTA, which includes a higher variety of poses and denser crowds, further improves the accuracy on CrowdPose.…”

A Survey of Image Synthesis Methods for Visual Machine Learning

“…Fabbri et al . [FLC*18] tackle the problem of multi‐person pose estimation and tracking by providing 10 million body poses collected from the Grand Theft Auto V (GTA‐V) video game [GTA]. They developed a game mod and created virtual scenes of crowds and pedestrian flow along with behaviour alterations (such as sitting and running).…”

“…Real‐time and non‐procedural physically based modelling based data generation approaches have been mainly the source of annotated data for this task. Methods involving GTA‐V extracted video sequences and Unity developed virtual worlds provide training data for multi‐object and multi‐person tracking [GWCV16, RHK17, FLC*18].…”

Explicit Spatiotemporal Joint Relation Learning for Tracking Human Pose

“…Relationships between joints are also considered by methods for multiperson tracking and grouping. For the multi-person tracking problem, the goal is to estimate the pose of all persons appearing in the video and assign a unique identity to each person [36,25,80,81,17,32,14]. These approaches differ from one another in their choice of metrics and features used for measuring similarity between a pair of poses and in their algorithms for joint identity assignment.…”