Two-person interaction recognition via spatial multiple instance embedding

Abstract

a b s t r a c tIn this work, we look into the problem of recognizing two-person interactions in videos. Our method integrates multiple visual features in a weakly supervised manner by utilizing an embedding-based multiple instance learning framework. In our proposed method, first, several visual features that capture the shape and motion of the interacting people are extracted from each detected person region in a video. Then, twoperson visual descriptors are formed. Since the relative spatial locations of interacting people are likely to complement the visual descriptors, we propose to use spatial multiple instance embedding, which implicitly incorporates the distances between people into the multiple instance learning process. Experimental results on two benchmark datasets validate that using two-person visual descriptors together with spatial multiple instance learning offers an effective way for inferring the type of the interaction.

Analyzing human–human interactions: A survey

“…Sefidgar et al [113] use the same reasoning to create a model based on discriminative key frames and consider their relative distance and timing within the interaction. Sener and Ikizler-Cinbis [116] formulate interaction detection as a multiple-instance learning problem to select relevant frames, because not all frames in an interaction are considered informative.…”

Hands-on: deformable pose and motion models for spatiotemporal localization of fine-grained dyadic interactions

“…Yang et al [20] improve classification in these cases by building detectors for various types of physical interactions such as hand-hand and hand-shoulder touches. The relative distance between individuals has been further explored by Sener and İkizler [21], who formulate interaction detection as a multiple-instance learning problem because not all frames in an interaction are considered informative. Sefidgar et al [22] use the same reasoning to create a model based on discriminative key frames and consider their relative distance and timing within the interaction.…”

“…The torso parts of both actors are connected through a virtual root part of the graph. This part does not have an associated part detector but it allows us to model relative distances between people, similar to Patron-Perez et al [5] and Sener and İkizler [21]. We enforce that the size of the virtual root part is equal to the size of the entire dyad of bodies, regardless of the locations and sizes of the associated part detectors.…”

“…The following interactions are performed: handshake, hug, kick, point, punch, and push. No pose data is available and we use the bounding box data from [21] as ground truth.…”

Local Temporal Block Difference Pattern for Action Recognition in Surveillance Videos using Tree Based Classifiers

“…Assorted approaches for action recognition in HAR have been proposed by Marín-Jiménez MJ, Yeguas (2013) [16], where Spatio-Temporal Interest points (STIP) are identified to enable the action recognition better.Spatio-Temporal Interest points were extracted from video sequences through Harris3D actions in videos where different types of low-and middle-level features have been used to predict action. Sener F, Ikizler-Cinbis, (2015) [17] elaborated the interaction between two people in action recognition. Tracing interacting persons, acquiring visual descriptors together with the distance between the actors, in the multiple learning process help to recognize the action.…”