Robust motion in-betweening

Abstract

In this work we present a novel, robust transition generation technique that can serve as a new tool for 3D animators, based on adversarial recurrent neural networks. The system synthesises high-quality motions that use temporally-sparse keyframes as animation constraints. This is reminiscent of the job of in-betweening in traditional animation pipelines, in which an animator draws motion frames between provided keyframes. We first show that a state-of-the-art motion prediction model cannot be easily converted into a robust transition generator when only adding conditioning information about future keyframes. To solve this problem, we then propose two novel additive embedding modifiers that are applied at each timestep to latent representations encoded inside the network's architecture. One modifier is a time-to-arrival embedding that allows variations of the transition length with a single model. The other is a scheduled target noise vector that allows the system to be robust to target distortions and to sample different transitions given fixed keyframes. To qualitatively evaluate our method, we present a custom MotionBuilder plugin that uses our trained model to perform in-betweening in production scenarios. To quantitatively evaluate performance on transitions and generalizations to longer time horizons, we present well-defined in-betweening benchmarks on a subset of the widely used Human3.6M dataset and on LaFAN1, a novel high quality motion capture dataset that is more appropriate for transition generation. We are releasing this new dataset along with this work, with accompanying code for reproducing our baseline results.

Learning a deep motion interpolation network for human skeleton animations

“…Harvey et al 11 propose a recurrent transition networks based architecture to generate transition animation from past frames along with a target keyframe, which they called ERD-quaternion velocity network (ERD-QV). When generating multiple transitions from several keyframes, the model is simply applied in series, using its last generated frames as past context for the next sequence.…”

Convolutional Autoencoders for Human Motion Infilling

“…However, in [14] the authors mention that post-optimization is used to close discontinuities in the transition from the last predicted frame to the first known frame of the target sequence. [15] is a recent follow-up work. While demonstrating impressive results, it mentions the use of inverse kinematics as a post-processing step.…”

“…A variant of motion infilling, known as keyframing, is a long-standing problem in the computer graphics community with the latest addition provided by [15]. Keyframing aims at reducing the amount of tedious manual labor for artists to create life-like animations without relinquishing control over the resulting animation.…”

“…For example, Berglund et al [6] show how a bi-directional structure can be used for the task, but the supported gap size is fixed and short. Very recently, Harvey et al [15] introduced time-to-arrival embeddings to a recurrent structure to support variable gap sizes. However, the authors also report the use of inverse kinematics to post-process the model's outputs.…”

Gap Reconstruction in Optical Motion Capture Sequences Using Neural Networks

“…In the work of Fragkiadaki et al [ 31 ], an encoder–recurrent-decoder (ERD) was proposed, employing long-short term memory (LSTM) as a recurrent layer. A similar approach (ERD) was proposed by Harvey et al [ 32 ] for in-between motion generation on the basis of asmall amount of keyframes. Mall et al [ 33 ] modified the ERD and proposed an encoder–bidirectional-filter (EBF) based on the bidirectional LSTM (BILSTM).…”