High Five: Recognising human interactions in TV shows

Abstract

In this paper we address the problem of recognising interactions between two people in realistic scenarios for video retrieval purposes. We develop a per-person descriptor that uses attention (head orientation) and the local spatial and temporal context in a neighbourhood of each detected person. Using head orientation mitigates camera view ambiguities, while the local context, comprised of histograms of gradients and motion, aims to capture cues such as hand and arm movement. We also employ structured learning to capture spatial relationships between interacting individuals.We train an initial set of one-vs-the-rest linear SVM classifiers, one for each interaction, using this descriptor. Noting that people generally face each other while interacting, we learn a structured SVM that combines head orientation and the relative location of people in a frame to improve upon the initial classification obtained with our descriptor. To test the efficacy of our method, we have created a new dataset of realistic human interactions comprised of clips extracted from TV shows, which represents a very difficult challenge. Our experiments show that using structured learning improves the retrieval results compared to using the interaction classifiers independently.

Human Instance Segmentation from Video using Detector-based Conditional Random Fields

“…Ever since computer vision researchers embarked on developing algorithms for complete scene understanding, understanding human activities has been an important goal [5,17]. More importantly, human activity recognition has found a niche in security and video surveillance and most of the related algorithms [5] depend on recovering human shapes and structures in the images.…”

Recognizing Complex Human Activities via Crowd Context

“…Research by Ryoo and Aggarwal [43], Yao et al [54] and Patron et al [40] goes beyond single-person activity understanding and propose methods for modeling interactions between pairs of individuals. The extension to activities that involve more than two individuals has been investigated in a number of works including [2,25,44].…”

3D GLOH features for human action recognition

“…To investigate the effectiveness of our approach for action recognition, experiments were conducted on three video datasets, namely UCF Sports [15], TV-Human Interaction [16] and KTH [17]. These datasets differ in several aspects such as recording conditions, scenarios, number of actors in video.…”