Real-Time Marker-Less Multi-person 3D Pose Estimation in RGB-Depth Camera Networks

Abstract

This paper proposes a novel system to estimate and track the 3D poses of multiple persons in calibrated RGB-Depth camera networks. The multi-view 3D pose of each person is computed by a central node which receives the single-view outcomes from each camera of the network. Each single-view outcome is computed by using a CNN for 2D pose estimation and extending the resulting skeletons to 3D by means of the sensor depth. The proposed system is marker-less, multiperson, independent of background and does not make any assumption on people appearance and initial pose. The system provides real-time outcomes, thus being perfectly suited for applications requiring user interaction. Experimental results show the effectiveness of this work with respect to a baseline multi-view approach in different scenarios. To foster research and applications based on this work, we released the source code in OpenPTrack, an open source project for RGB-D people tracking.

Real-time Tracking-by-Detection of Human Motion in RGB-D Camera Networks

“…For all the six sequences, we computed the same performance metrics (average joint displacement error and standard deviation) on the estimates provided by other state-of-the-art approaches using as input exactly the same data from all the four available Kinects. The other comparison methods have been: (i) OpenPose [14] enriched with the data association and depth inference algorithms, (ii) moving average filtering (MAF), a common baseline approach already described in other similar state-of-the-art works such as [3], [19], and (iii) the standard version of OpenPTrack [3]. The obtained results are reported in Table I.…”

“…However, the employment of a single sensor limits the reliability of the estimates, due to the fact that they are generally affected by occlusions and field-of-view limitations. A common solution seems to be connecting several cameras to form a common network [3], [20]. One of the biggest challenges when exploiting a multiple-camera network consists in the methodology used to merge information from different sensors.…”

“…Although a variety of different solutions to estimate 3D body poses from a single RGB-D sensor exists, in this work we used the one described in [3] which extends the singleview approach described in [14]. Despite our work not being constrained by the specific run-time single-view detection approach, the rationale behind our choice stands in its general applicability since its performances are independent from both the number of people to be tracked and the movements they perform.…”

“…where w ≥ 1 ∈ R. 3 In this way, the joint-specific threshold is potentially capable of slowly adapting to the changes in joint speed. If the new detection distance for the m-th joint d m,t is larger than the just computed threshold th m,t , the detection is not directly rejected, but the corresponding measurement noise variance, after the joint confidence adaptation, is updated as follow:…”

“…In details, with respect to our previous work [3], this paper introduces four novel elements: (i) a new implementation of the Kalman filter considering in its state all the joint positions and velocities of the skeleton model (Sec. III-B), (ii) a joint confidence feedback to adjust the variance of the measurement noise process of the Kalman filter according to the confidence level associated to each single-view detection (Sec.…”