Efficient Online Multi-Person 2D Pose Tracking With Recurrent Spatio-Temporal Affinity Fields

Raaj, Yaadhav; Idrees, Haroon; Hidalgo, Gines; Sheikh, Yaser

doi:10.1109/cvpr.2019.00475

Cited by 118 publications

(81 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Finally, a particularly important component to develop in future work with SLEAP will be to incorporate learnable tracking to enable the pose estimation models to better take advantage of temporal context. For example, the PAF representation could be extended to the time domain [27]. The top-down approach can also combine detection and tracking [31], although this requires sets of contiguous ground-truth frames which greatly increases the time and effort required for labeling.…”

Section: Discussionmentioning

confidence: 99%

SLEAP: Multi-animal pose tracking

Pereira

Tabris

et al. 2020

Preprint

View full text Add to dashboard Cite

The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation to quantify and model natural animal behavior. This has led to important advances in deep learning-based markerless pose estimation that have been enabled in part by the success of deep learning for computer vision applications. Here we present SLEAP (Social LEAP Estimates Animal Poses), a framework for multi-animal pose tracking via deep learning. This system is capable of simultaneously tracking any number of animals during social interactions and across a variety of experimental conditions. SLEAP implements several complementary approaches for dealing with the problems inherent in moving from single-to multi-animal pose tracking, including configurable neural network architectures, inference techniques, and tracking algorithms, enabling easy specialization and tuning for particular experimental conditions or performance requirements. We report results on multiple datasets of socially interacting animals (flies, bees, and mice) and describe how dataset-specific properties can be leveraged to determine the best configuration of SLEAP models. Using a high accuracy model (<2.8 px error on 95% of points), we were able to track two animals from full size 1024 × 1024 pixel frames at up to 320 FPS. The SLEAP framework comes with a sophisticated graphical user interface, multi-platform support, Colab-based GPU-free training and inference, and complete tutorials available, in addition to the datasets, at sleap.ai.

show abstract

Section: Discussionmentioning

confidence: 99%

SLEAP: Multi-animal pose tracking

Pereira

Tabris

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Then, we extract the feature data of each canvas using a PoseNet model that can be called with the browser [ 22 , 53 ]. In recent years, other pose estimate models such as the ones described in [ 54 , 55 , 56 ] have demonstrated potential for use in smart homes. To improve the process of continuously extracting feature data, we introduce a method to implement an offline PoseNet model.…”

Section: Methodsmentioning

confidence: 99%

Nonintrusive Fine-Grained Home Care Monitoring: Characterizing Quality of In-Home Postural Changes Using Bone-Based Human Sensing

Chen

Saiki

Nakamura

2020

Sensors

View full text Add to dashboard Cite

In contrast to the physical activities of able-bodied people at home, most people who require long-term specific care (e.g., bedridden patients and patients who have difficulty walking) usually show more low-intensity slow physical activities with postural changes. Although the existing devices can detect data such as heart rate and the number of steps, they have been increasing the physical burden relying on long-term wearing. The purpose of this paper is to realize a noninvasive fine-grained home care monitoring system that is sustainable for people requiring special care. In the proposed method, we present a novel technique that integrates inexpensive camera devices and bone-based human sensing technologies to characterize the quality of in-home postural changes. We realize a local process in feature data acquisition once per second, which extends from a computer browser to Raspberry Pi. Our key idea is to regard the changes of the bounding box output by standalone pose estimation models in the shape and distance as the quality of the pose conversion, body movement, and positional changes. Furthermore, we use multiple servers to realize distributed processing that uploads data to implement home monitoring as a web service. Based on the experimental results, we conveyed our findings and advice to the subject that include where the daily living habits and the irregularity of home care timings needed improvement.

show abstract

“…It uses an adaptive background subtraction in order to identify foreground regions for catching users' movements. In [29], authors present an online approach to simultaneously detect 2D poses of multiple people in a video sequence. They exploit Part Affinity Field (PAF) representations designed for static images, and they propose an architecture that can encode Spatio-Temporal Affinity Fields (STAF) across a video sequence.…”

Section: Related Workmentioning

confidence: 99%

Perceiving space through sound: mapping human movements into MIDI

Breve¹,

Cirillo²,

Desiato³

et al. 2020

International Conferences on Distributed Multimedia Systems

View full text Add to dashboard Cite

Gestural expressiveness plays a fundamental role in the interaction with people, environments, animals, things, and so on. Thus, several emerging application domains would exploit the interpretation of movements to support their critical designing processes. To this end, new forms to express the people's perceptions could help their interpretation, like in the case of music. This paper presents a novel algorithm for mapping human movements into MIDI music. The algorithm has been implemented in a system that integrates a module for real-time tracking of movements through a sample-based synthesizer using different types of filters to modulate frequencies. The system has been evaluated through a user study, in which several users have participated to a room experience, yielding significant results about their perceptions with respect to the environment they were immersed.

show abstract

Efficient Online Multi-Person 2D Pose Tracking With Recurrent Spatio-Temporal Affinity Fields

Cited by 118 publications

References 33 publications

SLEAP: Multi-animal pose tracking

SLEAP: Multi-animal pose tracking

Nonintrusive Fine-Grained Home Care Monitoring: Characterizing Quality of In-Home Postural Changes Using Bone-Based Human Sensing

Perceiving space through sound: mapping human movements into MIDI

Contact Info

Product

Resources

About