Deep Representation Learning for Human Motion Prediction and Classification

Abstract

Generative models of 3D human motion are often restricted to a small number of activities and can therefore not generalize well to novel movements or applications. In this work we propose a deep learning framework for human motion capture data that learns a generic representation from a large corpus of motion capture data and generalizes well to new, unseen, motions. Using an encoding-decoding network that learns to predict future 3D poses from the most recent past, we extract a feature representation of human motion. Most work on deep learning for sequence prediction focuses on video and speech. Since skeletal data has a different structure, we present and evaluate different network architectures that make different assumptions about time dependencies and limb correlations. To quantify the learned features, we use the output of different layers for action classification and visualize the receptive fields of the network units. Our method outperforms the recent state of the art in skeletal motion prediction even though these use action specific training data. Our results show that deep feedforward networks, trained from a generic mocap database, can successfully be used for feature extraction from human motion data and that this representation can be used as a foundation for classification and prediction.

Structured Prediction Helps 3D Human Motion Modelling

“…Bütepage et al [2] propose to encode poses with a hierarchy of dense layers following the kinematic chain starting from the end-effectors (dubbed H-TE), which is similar to our SP-layer. In contrast to this work, H-TE operates on the input rather than the output, and has only been demonstrated with non-recurrent networks when using 3D positions to parameterize the poses.…”

“…Bütepage et al [2,3] and Holden et al [10] convert the data directly to 3D joint positions. These works do not use recurrent structures, which necessitates the extraction of fixed-size, temporal windows for training.…”

“…These works do not use recurrent structures, which necessitates the extraction of fixed-size, temporal windows for training. [2] and [10] focus on learning of latent representations, which are shown to be helpful for various tasks, such as denoising, forecasting, or motion generation along a given trajectory [9]. [3] extends [2] by applying a conditional variational autoencoder (VAE) to the task of online motion prediction in human-robot interactions.…”

Predicting 3D Human Dynamics From Video

“…Earlier works model the synthesis of human motion using techniques such as Hidden Markov Models [7], linear dynamical systems [40], bilinear spatiotemporal basis models [2], and Gaussian process latent variable models [45,56] and other variants [22,55]. More recently, there are deep learning-based approaches that use recurrent neural networks (RNNs) to predict 3D future human motion from past 3D human skeletons [18,24,9,32,47]. All of these approaches operate in the domain where the inputs are 3D past motion capture sequences.…”

Deep convolutional and recurrent neural networks for cell motility discrimination and prediction

“…Recurrent neural networks have been effectively utilized for sequence prediction in multiple fields [74], [75], [76], [77]. We adapted the convolutional RNN autoencoder model to a sequence prediction model by removing the pooling layers and fully-connected layers and altering the number of nodes in the central LSTM layer (Fig.…”