Spatio-Temporal Manifold Learning for Human Motions via Long-Horizon Modeling

Abstract

Data-driven modeling of human motions is ubiquitous in computer graphics and computer vision applications, such as synthesizing realistic motions or recognizing actions. Recent research has shown that such problems can be approached by learning a natural motion manifold using deep learning on a large amount data, to address the shortcomings of traditional data-driven approaches. However, previous deep learning methods can be sub-optimal for two reasons. First, the skeletal information has not been fully utilized for feature extraction. Unlike images, it is difficult to define spatial proximity in skeletal motions in the way that deep networks can be applied for feature extraction. Second, motion is time-series data with strong multi-modal temporal correlations between frames. On the one hand, a frame could be followed by several candidate frames leading to different motions; on the other hand, long-range dependencies exist where a number of frames in the beginning correlate to a number of frames later. Ineffective temporal modeling would either under-estimate the multi-modality and variance, resulting in featureless mean motion or over-estimate them resulting in jittery motions, which is a major source of visual artifacts. In this paper, we propose a new deep network to tackle these challenges by creating a natural motion manifold that is versatile for many applications. The network has a new spatial component for feature extraction. It is also equipped with a new batch prediction model that predicts a large number of frames at once, such that long-term temporally-based objective functions can be employed to correctly learn the motion multi-modality and variances. With our system, long-duration motions can be predicted/synthesized using an open-loop setup where the motion retains the dynamics accurately. It can also be used for denoising corrupted motions and synthesizing new motions with given control signals. We demonstrate that our system can create superior results comparing to existing work in multiple applications.

Dynamic Future Net

“…We use a 8 to 20-frame motion prefix to start motion generation to get 900 frames (dfn_run2box_2char and dfn_boxing_3char in the video). The motion stability indicates that DFN does not suffer from the problem of cumulative error that is common in time-series generation [41]. Given the same prefix, the diversity is shown in their transitions between different postures (short-term) and different actions (long-term).…”

“…Despite the improved accuracy and lowered costs of motion capture systems, it is still highly desirable to make full use of existing data to generate diversified new data. One key challenge in motion generation is the dynamics modelling, where it has been shown that a latent space can be found due to the high coordination of body motions [21,36,41]. However, as much as the spatial aspect is studied, dynamics modelling, especially with the aim of diversified motion generation, still remains to be an open problem.…”

“…Tradition methods model them by Finite State Machines with carefully organized data [33], which have limited model capacities for large amounts of data and require extensive preprocessing work. New deep learning methods either ignore them [21] or do not explicitly model them [41]. Very recently, dynamics modelling for diversified generation has just been investigated [42], but only from the perspective of the overall dynamics, rather than the detailed short/long term stochasticity.…”

“…The state-wise bidirectional modelling (extending into both the past and future) separates DFN from existing methods and endows us with the ability of modelling the short-term (next-frame) randomness and long-term (next-action) randomness. Last, for inference purposes, we propose new loss functions based on distributional similarities as opposed to pointwise estimation [41,44], which captures the dynamics accurately but also keep the randomness that is crucial for diversified motion generation.…”

Automatic Sign Dance Synthesis from Gesture-based Sign Language

“…Zhou et al [36] proposed a new recurrent neural network to synthesize arbitrary motions with highly complex styles. Wang et al [33] included the concept of temporal prediction in recurrent neural network to create a more stable motion manifold. Lee et al [21] synthesized human-object interaction by introducing motion grammar, a syntax to describe interaction, into a deep neural network.…”