Behavior Discovery and Alignment of Articulated Object Classes from Unstructured Video

Abstract

We propose an automatic system for organizing the content of a collection of unstructured videos of an articulated object class (e.g., tiger, horse). By exploiting the recurring motion patterns of the class across videos, our system: (1) identifies its characteristic behaviors, and (2) recovers pixel-to-pixel alignments across different instances. Our system can be useful for organizing video collections for indexing and retrieval. Moreover, it can be a platform for learning the appearance or behaviors of object classes from Internet video. Traditional supervised techniques cannot exploit this wealth of data directly, as they require a large amount of time-consuming manual annotations. The behavior discovery stage generates temporal video intervals, each automatically trimmed to one instance of the discovered behavior, clustered by type. It relies on our novel motion representation for articulated motion based on the displacement of ordered pairs of trajectories. The alignment stage aligns hundreds of instances of the class to a great accuracy despite considerable appearance variations (e.g., an adult tiger and a cub). It uses a flexible thin plate spline deformation model Google, 1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA that can vary through time. We carefully evaluate each step of our system on a new, fully annotated dataset. On behavior discovery, we outperform the state-of-the-art improved dense trajectory feature descriptor. On spatial alignment, we outperform the popular SIFT Flow algorithm.

Self-supervised 3D Human Pose Estimation from a Single Image

“…We train the model with a dataset of video frames depicting full-body horses. First, we select the horse subset from the latest version of the TigDog dataset [9]. We use video frames for all the horse sequences in the dataset, discarding video frames showing partially visible horses.…”

“…We collect frames containing complete horses from about 60 videos, representing 47k frames (plus around 6k from [9]). Note that this dataset is relatively small compared to what is required for training human pose estimation models -our horse dataset is only 1.3% of the size of the Human3.6M dataset [13] and 3.6% of the size of the MPI-INF-3DHP dataset [21].…”

Discovering thematic visual objects in unconstrained videos

“…The existing action co-localization works have mainly focused on two scenarios, i.e., co-localization in pairs of videos [18,38,97] and weakly supervised action co-localization with video level labels [20,21,92,93,108,134,143]. Few of these works have considered a fully unconstrained scenario like us, i.e., the numbers and types of common actions are unknown in advance and each video may contain zero, one or several common actions.…”

“…It proposes a feature to describe pairs of trajectories in order to better characterize articulated object motion. [21] extends [20] to also recovers the spatial alignment among different instances of the same behavior using a Thin Plate Spline deformation model. [147] proposes a method to mine recurring events in a collection of videos.…”