Learning Compositional Representation for 4D Captures with Neural ODE

Abstract

Learning based representation has become the key to the success of many computer vision systems. While many 3D representations have been proposed, it is still an unaddressed problem for how to represent a dynamically changing 3D object. In this paper, we introduce a compositional representation for 4D captures, i.e. a deforming 3D object over a temporal span, that disentangles shape, initial state, and motion respectively. Each component is represented by a latent code via a trained encoder. To model the motion, a neural Ordinary Differential Equation (ODE) is trained to update the initial state conditioned on the learned motion code, and a decoder takes the shape code and the updated pose code to reconstruct 4D captures at each time stamp. To this end, we propose an Identity Exchange Training (IET) strategy to encourage the network to learn effectively decoupling each component. Extensive experiments demonstrate that the proposed method outperforms existing stateof-the-art deep learning based methods on 4D reconstruction, and significantly improves on various tasks, including motion transfer and completion.

Neural Schrödinger Equation: Physical Law as Deep Neural Network

“…The empirical study of the neural ODE on the image classification task are investigated in [74]. Applications of neural-ODEs to several actual tasks have been demonstrated in [75]- [77].…”