Efficient Neural Networks for Real-time Motion Style Transfer

Abstract

Style is an intrinsic, inescapable part of human motion. It complements the content of motion to convey meaning, mood, and personality. Existing state-of-the-art motion style methods require large quantities of example data and intensive computational resources at runtime. To ensure output quality, such style transfer applications are often run on desktop machine with GPUs and significant memory. In this paper, we present a fast and expressive neural network-based motion style transfer method that generates stylized motion with quality comparable to the state of the art method, but uses much less computational power and a much smaller memory footprint. Our method also allows the output to be adjusted in a latent style space, something not offered in previous approaches. Our style transfer model is implemented using three multi-layered networks: a pose network, a timing network and a foot-contact network. A one-hot style vector serves as an input control knob and determines the stylistic output of these networks. During training, the networks are trained with a large motion capture database containing heterogeneous actions and various styles. Joint information vectors together with one-hot style vectors are extracted from motion data and fed to the networks. Once the network has been trained, the database is no longer needed on the device, thus removing the large memory requirement of previous motion style methods. At runtime, our model takes novel input and allows real-valued numbers to be specified in the style vector, which can be used for interpolation, extrapolation or mixing of styles. With much lower memory and computational requirements, our networks are efficient and fast enough for real-time use on mobile devices. Requiring no information about future states, the style transfer can be performed in an online fashion. We validate our result both quantitatively and perceptually, confirming its effectiveness and improvement over previous approaches.

Adult2child: Motion Style Transfer using CycleGANs

“…Our work shows that CycleGAN's can be adapted to transfer both pose style and timing in a single network, in contrast to other recent works which do not handle temporal differences in style Smith et al 2019]. Another advantage compared to the traditional style translation methods that use time-warping [Hsu et al 2005;Smith et al 2019], is that our algorithm can work on unaligned data and learn the temporal mapping from adult motions to child motions. Dynamic time warping is a time-consuming process, and importantly, it relies on having a beginning and and end frame that match.…”

“…Recently, Smith et al [2019] introduced a computationally efficient method using three multi-layer neural networks that motion to be adjusted in a latent style space, thus achieving real-time style transfer with low memory requirements. The main limitation in their approach is that they have a separate network to learn the timing, and this timing adjustment is applied as a post-process.…”

“…However, retargeting mostly involves changes in the dimensions of limbs, so mapping adult motion directly on child characters fails to transfer the style and nuances of the children motion such as speed and variability. Style translation, that is, learning a mapping between two labeled motion capture sequences, has been extensively studied, starting with approaches by Brand and Hertzmann [2000] and Gleicher [1998] to recent advances made by deep neural networks Du et al 2019a;Holden et al 2017Holden et al , 2016Holden et al , 2015Mason et al 2018;Smith et al 2019].…”

Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio

“…Synthesizing motion with specific movement style has been studied in a large body of prior works [44,39,50,6,15]. Most methods formulate the problem as transferring a specific motion style to an input motion [53,44], or transferring the motion from one character to another, commonly referred as motion retargeting [14,8,46]. Recent approaches explored deep reinforcement learning to model physics-based locomotion with a specific style [38,33,39].…”

Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model

“…Style Transfer Approaches : Motivated by recent advances in style transfer in images and videos, style transfer techniques were exploited to transfer the style of one animation clip to another [SCNW19, AWL*20]. While this method generates natural motion sequences with the desired style, these approaches cannot be directed by other control signals.…”