Learning to Identify Physical Parameters from Video Using Differentiable Physics

Abstract

Video representation learning has recently attracted attention in computer vision due to its applications for activity and scene forecasting or vision-based planning and control. Video prediction models often learn a latent representation of video which is encoded from input frames and decoded back into images. Even when conditioned on actions, purely deep learning based architectures typically lack a physically interpretable latent space. In this study, we use a differentiable physics engine within an action-conditional video representation network to learn a physical latent representation. We propose supervised and self-supervised learning methods to train our network and identify physical properties. The latter uses spatial transformers to decode physical states back into images. The simulation scenarios in our experiments comprise pushing, sliding and colliding objects, for which we also analyze the observability of the physical properties. In experiments we demonstrate that our network can learn to encode images and identify physical properties like mass and friction from videos and action sequences in the simulated scenarios. We evaluate the accuracy of our supervised and self-supervised methods and compare it with a system identification baseline which directly learns from state trajectories. We also demonstrate the ability of our method to predict future video frames from input images and actions. Electronic supplementary material The online version of this chapter (10.1007/978-3-030-71278-5_4) contains supplementary material, which is available to authorized users.

ϕ-SfT: Shape-from-Template with a Physics-Based Deformation Model

“…In contrast to our method, they additionally require depth inputs for a strong 3D cue and do not recover local surface deformations. The idea of combining a differentiable physics and a differentiable graphics engine has been recently explored in contexts different from ours, i.e., estimation of material properties and visuomotor control [21,23,31].…”

Vision-based system identification and 3D keypoint discovery using dynamics constraints

“…Unsupervised system identification from vision is a recent area of research that removes the requirements for trajectory data, with approaches including unsupervised physical parameter estimation [24,31,40], structured latent space learning [19,25,32], and Hamiltonian/Lagrangian learning [18,51,58]. Unfortunately, these approaches are still relatively limited in the complexity of scene they can model, and typically restricted to toy problems and simulated environments.…”

“…Unfortunately, these approaches are still relatively limited in the complexity of scene they can model, and typically restricted to toy problems and simulated environments. In this work we aim to improve upon [24,31,40]'s limitation to simulated environments by performing physical parameter estimation on real dynamical scenes with distractors.…”

DiffSDFSim: Differentiable Rigid-Body Dynamics With Implicit Shapes

“…Weiss et al [32] estimate material properties of deformables by matching a differentiable deformable simulation to point cloud observations. In Kandukuri et al [15] a differentiable physics simulation based on [9] is embedded as layer into a deep neural network which infers the physical state from images and predicts the next states. Recently, [16] proposed gradSim, a framework that combines differentiable simulation and differentiable rendering for system identification from video and visual control.…”