A Game-Theoretic Probabilistic Approach for Detecting Conversational Groups

Abstract

A standing conversational group (also known as F-formation) occurs when two or more people sustain a social interaction, such as chatting at a cocktail party. Detecting such interactions in images or videos is of fundamental importance in many contexts, like surveillance, social signal processing, social robotics or activity classification. This paper presents an approach to this problem by modeling the socio-psychological concept of an F-formation and the biological constraints of social attention. Essentially, an F-formation defines some constraints on how subjects have to be mutually located and oriented while the biological constraints defines the plausible zone in which persons can interact. We develop a game-theoretic framework embedding these constraints, which is supported by a statistical modeling of the uncertainty associated with the position and orientation of people. First, we use a novel representation of the affinity between pairs of people expressed as a distance between distributions over the most plausible oriented region of attention.Additionally, we integrate temporal information over multiple frames to smooth noisy head orientation and pose estimates, solve ambiguous situations and establish a more precise social context. We do this in a principled way by using recent notions from multi-payoff evolutionary game theory. Experiments on several benchmark datasets consistently show the superiority of the proposed approach over state of the art and its robustness under severe noise conditions.Author has been partially supported by the European Commission under contract number FP7-ICT-600877 (SPENCER) and is affiliated with the Delft Data Science consortium.

A Hierarchical Deep Temporal Model for Group Activity Recognition

“…Secondly, they also use a probabilistic motion analysis to extract interesting spatio-temporal patterns for scenario recognition. Vascon et al [23] detect conversational groups in crowded scenes of people. The approach uses pairwise affinities between people based on pose and a game-theoretic clustering procedure.…”

F-Formations for Social Interaction in Simulation Using Virtual Agents and Mobile Robotic Telepresence Systems

“…These constraint-based formations are shown in Figure 3. Formations are considered very useful in analyzing and increasing the quality of interaction in social interactions [1,10,11], and a number of works [19][20][21][22][23][24][25] have proposed different methods to detect F-formations automatically. The Hough voting strategy (density estimation) was used to locate the O-space (see Figure 2a) by considering each person's position and head orientation in [19].…”

“…Frustum of attention to extract features from individuals and accordingly classify associates, singletons, and members of F-formations was used in [25]. Vascon et al [23] developed a game-theoretic model embedding the social-psychological concept of an F-formation and the biological constraints of social attention. They generated a frustum based on the position and orientation of each person and computed affinity to extract the F-formation.…”

SALSA: A Novel Dataset for Multimodal Group Behavior Analysis

“…Also, errors observed for head pose are considerably smaller than for body pose over all four camera views- this is because body pose classifiers are impeded by severe occlusions in crowded scenes. Precisely for this reason, previous works on F-formation detection from FCGs [30], [32], [59] have primarily employed head orientation, even though body pose has been widely acknowledged as the more reliable cue for determining interacting persons. We believe that devising a multimodal approach also employing IR and bluetooth-based sensors for body pose estimation would be advantageous as compared to a purely visual analysis, which was one of the primary motives for compiling the SALSA dataset.…”