Hands-on: deformable pose and motion models for spatiotemporal localization of fine-grained dyadic interactions

Abstract

We introduce a novel spatiotemporal deformable part model for the localization of fine-grained human interactions of two persons in unsegmented videos. Our approach is the first to classify interactions and additionally provide the temporal and spatial extent of the interaction in the video. To this end, our models contain part detectors that support different scales as well as different types of feature descriptors, which are combined in a single graph. This allows us to model the detailed coordination between people in terms of body pose and motion. We demonstrate that this helps to avoid confusions between visually similar interactions. We show that robust results can be obtained when training on small numbers of training sequences (5-15) per interaction class. We achieve AuC scores of 0.82 with an IoU of 0.3 on the publicly available ShakeFive2 dataset, which contains interactions that differ slightly in their coordination. To further test the generalization of our models, we perform cross-dataset experiments where we test on two other publicly available datasets: UT-Interaction and SBU Kinect. These experiments show that our models generalize well to different environments.

Analyzing human–human interactions: A survey

“…This approach can even be extended to deal with interactions for which no, or very little, training data is available. Specifically for human-human interactions, the coordination of pose and motion is crucial for distinguishing between subtly different classes [38]. Mid-level representations should take into account this coordination in both space and time, such as the distance and orientation between people, or the relative placement of a hand on the other's shoulder.…”

“…Especially in sustained or repeated social encounters, for example in public spaces, knowing the actors that interact would increase the efficacy of the analysis. A few works have addressed interaction detection (e.g., [38,135]) but usually in a two-step approach by first detection humans (e.g., [95]) and then considering their interactions. Especially in more crowded settings where partial occlusions are more common, such an approach is more likely to fail.…”

“…UT-Interaction is used as benchmark for many methodologies, ranging from bounding boxes techniques [89,121] to bags-of-visual-words [118,125]. Some works have also addressed the detection of interactions in both space and time [38].…”

“…The resulting volume is then segmented into supervoxels, to refine the person's outline in each frame, and classify the action. van Gemeren et al [38] use interaction-specific DPMs with poselet parts [10] to locate people in poses characteristic for a given interaction. Instead of encoding the orientation of (pairs of) limbs as poselets, DPMs can also include a larger number of articulations by using a mixture of parts [159].…”

“…These formulations work well for the representation of coarse movements, but finer-scale movements are difficult to model because the motion is not linked to specific parts of the body. An extension of DPMs to include part deformations not only in space and scale, but also in time is presented in van Gemeren et al [38]. This enables the detection, in both space and time, of interactions that are characterized by a single key pose or key movement phase.…”