OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association

Kreiss, S.; Bertoni, Lorenzo; Alahi, Alexandre

doi:10.1109/tits.2021.3124981

Cited by 88 publications

(48 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different methods have been proposed (14,(29)(30)(31). Affinity-based methods such as OpenPose (30) and PifPaf (16,32) predict the affinity between keypoints and use post processing to link keypoints of individuals. Others (29,31) apply associative embeddings (AE) to achieve keypoint grouping.…”

Section: Related Workmentioning

confidence: 99%

“…Our proposed models benefit from diverse data and potentially learns pose features that are agnostic to the species of the animals downstream without any labeled fine-tuning. Moreover, due to its improved quality of representation, such a model could be significantly more data efficient, and thus provide better pre-training weights than commonly used Im-ageNet pre-trained weights (e.g., (5,8,(14)(15)(16)).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SuperAnimal models pretrained for plug-and-play analysis of animal behavior

Ye¹,

Mathis²,

Mathis³

2022

Preprint

View full text Add to dashboard Cite

Animal pose estimation is critical in applications ranging from life science research, agriculture, to veterinary medicine. Compared to human pose estimation, the performance of animal pose estimation is limited by the size of available datasets and the generalization of a model across datasets. Typically different keypoints are labeled regardless of whether the species are the same or not, leaving animal pose datasets to have disjoint or partially overlapping keypoints. As a consequence, a model cannot be used as a plug-and-play solution across datasets. This reality motivates us to develop panoptic animal pose estimation models that are able to predict keypoints defined in all datasets. In this work we propose a simple yet effective way to merge differentially labeled datasets to obtain the largest quadruped and lab mouse pose dataset. Using a gradient masking technique, so called SuperAnimal-models are able to predict keypoints that are distributed across datasets and exhibit strong zero-shot performance. The models can be further improved by (pseudo) labeled fine-tuning. These models outperform ImageNet-initialized models.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

SuperAnimal models pretrained for plug-and-play analysis of animal behavior

Ye¹,

Mathis²,

Mathis³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Existing human key point extraction networks, such as OpenPose, AlphaPose, and OpenPifPaf [ 28 ], are extremely vulnerable to dense crowd occlusion or human self-occlusion, resulting in the low performance of human key point detectors.…”

Section: Fall Detection Algorithm For Shipboard Seafarers Based On Bl...mentioning

confidence: 99%

Fall Detection for Shipboard Seafarers Based on Optimized BlazePose and LSTM

Liu

et al. 2022

Sensors

View full text Add to dashboard Cite

Aiming to avoid personal injury caused by the failure of timely medical assistance following a fall by seafarer members working on ships, research on the detection of seafarer’s falls and timely warnings to safety officers can reduce the loss and severe consequences of falls to seafarers. To improve the detection accuracy and real-time performance of the seafarer fall detection algorithm, a seafarer fall detection algorithm based on BlazePose–LSTM is proposed. This algorithm can automatically extract the human body key point information from the video image obtained by the vision sensor, analyze its internal data correlation characteristics, and realize the process from RGB camera image processing to seafarer fall detection. This fall detection algorithm extracts the human body key point information through the optimized BlazePose human body key point information extraction network. In this section, a new method for human bounding-box acquisition is proposed. In this study, a head detector based on the Vitruvian theory was used to replace the pre-trained SSD body detector in the BlazePose preheating module. Simultaneously, an offset vector is proposed to update the bounding box obtained. This method can reduce the frequency of repeated use of the head detection module. The algorithm then uses the long short-term memory neural network to detect seafarer falls. After extracting fall and related behavior data from the URFall public data set and FDD public data set to enrich the self-made data set, the experimental results show that the algorithm can achieve 100% accuracy and 98.5% specificity for the seafarer’s falling behavior, indicating that the algorithm has reasonable practicability and strong generalization ability. The detection frame rate can reach 29 fps on a CPU, which can meet the effect of real-time detection. The proposed method can be deployed on common vision sensors.

show abstract

“…Anyway, for human detection, bounding boxes do not always provide enough information. For these reason, human pose or skeleton estimation have been proposed [3,13]. In this case, the aim is to predict the position of N human body joints.…”

Section: People Detectionmentioning

confidence: 99%

“…The proposal of Yang et al [26], called Multi-person Panoramic Localization and Tracking (MPLT), instead relies on four calibrated perspective cameras, positioned such that they cover a field of view of 360 • to simulate an omnidirectional camera. The OpenPifPaf framework [13] is used for skeleton detection and a pinhole camera model is used to map detections in the 3D space, under the assumption that the height of a person is almost constant. The tracking system models each couple of tracks with two costs, one associated with the appearance of the person and one associated with the distance of the trajectories estimated with a Kalman Filter.…”

Section: People Trackingmentioning

confidence: 99%

People Tracking in Panoramic Video for Guiding Robots

Bacchin¹,

Berno²,

Menegatti³

et al. 2022

Preprint

View full text Add to dashboard Cite

A guiding robot aims to effectively bring people to and from specific places within environments that are possibly unknown to them. During this operation the robot should be able to detect and track the accompanied person, trying never to lose sight of her/him. A solution to minimize this event is to use an omnidirectional camera: its 360 • Field of View (FoV) guarantees that any framed object cannot leave the FoV if not occluded or very far from the sensor. However, the acquired panoramic videos introduce new challenges in perception tasks such as people detection and tracking, including the large size of the images to be processed, the distortion effects introduced by the cylindrical projection and the periodic nature of panoramic images. In this paper, we propose a set of targeted methods that allow to effectively adapt to panoramic videos a standard people detection and tracking pipeline originally designed for perspective cameras. Our methods have been implemented and tested inside a deep learning-based people detection and tracking framework with a commercial 360 • camera. Experiments performed on datasets specifically acquired for guiding robot applications and on a real service robot show the effectiveness of the proposed approach over other state-of-the-art systems. We release with this paper the acquired and annotated datasets and the open-source implementation of our method.

show abstract

OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association

Cited by 88 publications

References 63 publications

SuperAnimal models pretrained for plug-and-play analysis of animal behavior

SuperAnimal models pretrained for plug-and-play analysis of animal behavior

Fall Detection for Shipboard Seafarers Based on Optimized BlazePose and LSTM

People Tracking in Panoramic Video for Guiding Robots

Contact Info

Product

Resources

About