A Quadruple Diffusion Convolutional Recurrent Network for Human Motion Prediction

Abstract

Recurrent neural network (RNN) has become popular for human motion prediction thanks to its ability to capture temporal dependencies. However, it has limited capacity in modeling the complex spatial relationship in the human skeletal structure. In this work, we present a novel diffusion convolutional recurrent predictor for spatial and temporal movement forecasting, with multi-step random walks traversing bidirectionally along an adaptive graph to model interdependency among body joints. In the temporal domain, existing methods rely on a single forward predictor with the produced motion deflecting to the drift route, which leads to error accumulations over time. We propose to supplement the forward predictor with a forward discriminator to alleviate such motion drift in the long term under adversarial training. The solution is further enhanced by a backward predictor and a backward discriminator to effectively reduce the error, such that the system can also look into the past to improve the prediction at early frames. The two-way spatial diffusion convolutions and two-way temporal predictors together form a quadruple network. Furthermore, we train our framework by modeling the velocity from observed motion dynamics instead of static poses to predict future movements that effectively reduces the discontinuity problem at early prediction. Our method outperforms the state of the arts on both 3D and 2D datasets, including the Human3.6M, CMU Motion Capture and Penn Action datasets. The results also show that our method correctly predicts both high-dynamic and low-dynamic moving trends with less motion drift.

Emotion Transfer for 3D Hand and Full Body Motion Using StarGAN

“…Since 2D Convolution focuses on local neighbours (i.e., image pixels nearby) only, using the sequential ordering method can result in sub-optimal results when representing the tree-like skeletal structure for full-body and hand motions as the neighbouring joints are not necessarily close-by after converting into the image representation. While encouraging results are obtained in this study, we will explore the use of other approaches to better represent motions in the StarGAN framework in the future, such as Graph Convolutional Networks (GCNs) [37] and its variants [38] which demonstrated better performance in modelling human-like skeletal motions.…”

Multi-Task Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising

“…These methods can be categorized based on the network design [34]. Spatial-temporal RNN [37] has been proposed for human motion prediction that utilizes skeletal information for feature extraction. Batch prediction addresses the ineffective temporal modeling of motion multi-modality and variances, resulting in accurately predicting long-duration motions [50].…”

3D Human Motion Prediction: A Survey

“…Additionally, widespread recognition of the Temporal Convolution Networks (TCNs), made the CN-based methods more widely accepted for solving the sequential problem. These methods include: TE [9], PAML [29], CHA [49], LDR [17], MGCN [110], LTD [67], LDR [17], MoPredNet [107], TrajectoryCNN [53], NAT [44], C-seq2seq [45], Q-DCRN [73], MT-GCN [15], LMC [110], MST-GNN [47], DA-GNN [48], JDM [91].…”

“…CN-based methods TE [9] 2017 C-seq2seq [45] 2018 PAML [29] 2018 CHA [49] 2019 LTD [67] 2019 LDR [17] 2020 MGCN [110] 2020 MoPredNet [107] 2020 TrajectoryCNN [53] 2021 NAT [44] 2021 Q-DCRN [73] 2021 MT-GCN [15] 2021 LMC [110], 2021 MST-GNN [47] 2021 DA-GNN [48] 2021 JDM [91] 2021…”