Pose Estimation and Segmentation of People in 3D Movies

Abstract

We seek to obtain a pixel-wise segmentation and pose estimation of multiple people in a stereoscopic video. This involves challenges such as dealing with unconstrained stereoscopic video, non-stationary cameras, and complex indoor and outdoor dynamic scenes. The contributions of our work are two-fold: First, we develop a segmentation model incorporating person detection, pose estimation, as well as colour, motion, and disparity cues. Our new model explicitly represents depth ordering and occlusion. Second, we introduce a stereoscopic dataset with frames extracted from feature-length movies "StreetDance 3D" and "Pina". The dataset contains 2727 realistic stereo pairs and includes annotation of human poses, person bounding boxes, and pixel-wise segmentations for hundreds of people. The dataset is composed of indoor and outdoor scenes depicting multiple people with frequent occlusions. We demonstrate results on our new challenging dataset, as well as on the H2view dataset from (Sheasby et al. ACCV 2012).

Retrieving Similar Styles to Parse Clothing

“…Some previous clothing recognition work has used face detection to first find a person's head and torso and use this to bootstrap localization of given clothing items [33]. Also some approaches to the pose estimation problem itself have taken advantage of image segmentation [43], [44], [45] for improving performance. In this paper, we show empirical evidence that semantic clothing segmentation is beneficial to improving pose estimation.…”

Multiple Human Pose Estimation with Temporally Consistent 3D Pictorial Structures

“…Part-based models have proved to be a powerful solution for 2D pose estimation [1,4,10,24]. Among many of the proposed models, pictorial structures is the most widely used one [3,11,13].…”

“…However, each frame has been treated independently without considering the temporal component. Recently, a segmentation approach has been combined with a part-based model for estimating the human pose from stereo data [1]. Despite the available 3D information, the final pose is in the 2D space.…”

“…In every application, the human motion remains consistent over time. Different approaches have been proposed for multiple human pose estimation on 2D [1,4,10,24] or 3D space [5,6,14,20,21]. Nevertheless, the utility of temporal consistency for pose estimation has not been sufficiently addressed yet.…”

Parsing human skeletons in an operating room

“…Defining the human body as a constellation of parts has been proved to be the effective way for 2D pose estimation 1 http://campar.in.tum.de/Chair/MultiHumanOR [2,3,17,41]. The most notable part-based model are Pictorial Structures for 2D [4,18,20] or 3D body pose estimation [7,14].…”