Deep motifs and motion signatures

Abstract

Many analysis tasks for human motion rely on high-level similarity between sequences of motions, that are not an exact matches in joint angles, timing, or ordering of actions. Even the same movements performed by the same person can vary in duration and speed. Similar motions are characterized by similar sets of actions that appear frequently. In this paper we introduce motion motifs and motion signatures that are a succinct but descriptive representation of motion sequences. We first break the motion sequences to short-term movements called motion words, and then cluster the words in a high-dimensional feature space to find motifs. Hence, motifs are words that are both common and descriptive, and their distribution represents the motion sequence. To cluster words and find motifs, the challenge is to define an effective feature space, where the distances among motion words are semantically meaningful, and where variations in speed and duration are handled. To this end, we use a deep neural network to embed the motion words into feature space using a triplet loss function. To define a signature, we choose a finite set of motion-motifs, creating a bag-of-motifs representation for the sequence. Motion signatures are agnostic to movement order, speed or duration variations, and can distinguish fine-grained differences between motions of the same class. We illustrate examples of characterizing motion sequences by motifs, and for the use of motion signatures in a number of applications.

Motion Words: A Text-Like Representation of 3D Skeleton Sequences

“…As the recall is very important for the search task, the 1/5 multi-overlay vocabulary is now the best candidate that also outperforms the baseline case. Compared to the state-of-the art approaches [3,11] that employ the hard quantization, the proposed generalized MWs reach much better effectiveness, e.g., about 25% higher recall in the search application (increase from 44.21% to 55.62%).…”

“…We divide each action synthetically into a sequence of overlapping segments. As recommended in [3], we fix the segment length to 80 frames and the segment overlap to 64 frames, so the segments are shifted by 16 frames. This generates 28 k segments in total, with 12 segments per action on average.…”

“…However, indexing a large number of such very high-dimensional features is costly. To move towards more efficient processing, the approaches in [3,11] quantize the segment features using a single k-means clustering. However, with such simple quantization the border problem appears frequently, which decreases the effectiveness of applications with an increasing number of clusters (i.e.…”

“…Basic data quantization is usually performed by the k-means algorithm that divides the segment space into non-overlapping partitions [3,11,21]. Each partition can be assigned a one-dimensional identifier, which constitutes the motion word.…”

Digital Dance Ethnography

“…We describe how dance motion can be acquired and documented holistically so as to enable the extraction of semantic, cultural, and contextual correlations. We have defined a descriptive representation for motions based on signatures [5], which rely on the distribution of some narrow temporal overlapping feature descriptors, named motion and style words. Signatures are timescale and temporal-order invariant, capable of exploiting the contextual correlation between dances.…”