Neural state machine for character-scene interactions

Abstract

We propose Neural State Machine , a novel data-driven framework to guide characters to achieve goal-driven actions with precise scene interactions. Even a seemingly simple task such as sitting on a chair is notoriously hard to model with supervised learning. This difficulty is because such a task involves complex planning with periodic and non-periodic motions reacting to the scene geometry to precisely position and orient the character. Our proposed deep auto-regressive framework enables modeling of multi-modal scene interaction behaviors purely from data. Given high-level instructions such as the goal location and the action to be launched there, our system computes a series of movements and transitions to reach the goal in the desired state. To allow characters to adapt to a wide range of geometry such as different shapes of furniture and obstacles, we incorporate an efficient data augmentation scheme to randomly switch the 3D geometry while maintaining the context of the original motion. To increase the precision to reach the goal during runtime, we introduce a control scheme that combines egocentric inference and goal-centric inference. We demonstrate the versatility of our model with various scene interaction tasks such as sitting on a chair, avoiding obstacles, opening and entering through a door, and picking and carrying objects generated in real-time just from a single model.

Character controllers using motion VAEs

“…For task-specific computer animation, supervised learning of (direct-predictive) parametric motion models have seen much recent interest. Autoregressive DNN models can produce high-quality human variable-terrain locomotion , quadruped variable-terrain locomotion [Zhang et al 2018], and environment aware human locomotion [Starke et al 2019]. Similarly, a mix of data augmentation and flexible objective annotation ] can be used to learn an effective task-specific RNN model for human motion, as demonstrated on locomotion, basketball, and tennis.…”

Local motion phases for learning multi-contact character movements

“…Thanks to its nature, such models can be used for real-time character control, which is our target application. Human motion has been modelled with various time series models, such as conditional Restricted Bolzmann Machine (cRBM) [Taylor et al 2007], Gaussian processes (GP) [Wang et al 2008], recurrent neural networks [Fragkiadaki et al 2015;Harvey and Pal 2018;Li et al 2017;Villegas et al 2018], Phase Functioned Neural Networks (PFNN) [Holden et al 2017] and mixture-of-experts models [Starke et al 2019;Xia et al 2015;.…”

Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model

“…Mixture‐of‐Experts Approaches : Another strategy exploited in [SZKS19, SZKZ20, LZCVDP20] to address the problem of mean collapse in multi‐modal motion data is to use a Mixture‐of‐Experts (MoE) network where each expert is responsible for one mode in the training data. Though effective at mitigating mean collapse, the number of parameters in these networks increases with the number of experts.…”