Behavior-Driven Synthesis of Human Dynamics

Blattmann, Andreas; Milbich, Timo; Dorkenwald, Michael; Ommer, Björn

doi:10.1109/cvpr46437.2021.01206

Cited by 12 publications

(12 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other works rely on a low-dimensional, parametric representation e.g. keypoints to transfer motion between videos [1,5] or to synthesize videos based on action labels [82]. Given such assumptions, these works cannot be universally applied to arbitrary object categories and allow only for coarse control compared to our fine-grained, local object manipulations.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis

Blattmann

Milbich

Dorkenwald

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

View full text Add to dashboard Cite

How would a static scene react to a local poke? What are the effects on other parts of an object if you could locally push it? There will be distinctive movement, despite evident variations caused by the stochastic nature of our world. These outcomes are governed by the characteristic kinematics of objects that dictate their overall motion caused by a local interaction. Conversely, the movement of an object provides crucial information about its underlying distinctive kinematics and the interdependencies between its parts. This two-way relation motivates learning a bijective mapping between object kinematics and plausible future image sequences. Therefore, we propose iPOKE -invertible Prediction of Object Kinematics -that, conditioned on an initial frame and a local poke, allows to sample object kinematics and establishes a one-to-one correspondence to the corresponding plausible videos, thereby providing a controlled stochastic video synthesis. In contrast to previous works, we do not generate arbitrary realistic videos, but provide efficient control of movements, while still capturing the stochastic nature of our environment and the diversity of plausible outcomes it entails. Moreover, our approach can transfer kinematics onto novel object instances and is not confined to particular object classes. Our project page is available at https://bit.ly/3dJN4Lf.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Controlled Video Synthesis Model. The method of Hao et al [27] is implemented based on the official code 5 and the provided hyperparameters for all used datasets. We used their proposed procedure to construct the motion trajectories based on the same optical flow which was we used to train our own model.…”

Section: Appendix Preliminariesmentioning

confidence: 99%

iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis

Blattmann

Milbich

Dorkenwald

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

View full text Add to dashboard Cite

show abstract

“…To reduce complexity, existing learning based approaches often focus on modelling human dynamics using low-dimensional, parametric representations such as keypoints [1,87,3], thus preventing universal applicability. Moreover, as these approaches are either based on explicit action labels or require motion sequences as input, they cannot be applied to controlling single body parts.…”

Section: Related Workmentioning

confidence: 99%

“…We provide four videos (poking_plants_ [1][2][3][4].mp4) for the PokingPlants dataset showing distinct types of plants of substantially different shapes and appearances. Despite these large variances, our model generates realistic and appealing visualizations which are plausible responses to the poke.…”

Section: A1 Pokingplantsmentioning

confidence: 99%

“…In the realm of still images the interplay between object shape and appearance has been extensively studied, even allowing for controlled, global [56,54,28,22,21] and local [52,89] manipulation. Existing work on object dynamics, however, so far is addressed by either extrapolations of observed object motion [83,37,16,45,83,59,8,76] or only coarse control of predefined attributes such as explicit action labels [87] and imitation of previously observed holistic motion [1,20,3]. Directly controlling and, even further, interacting with objects on a local level however, so far, is a novel enterprise.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Understanding Object Dynamics for Interactive Image-to-Video Synthesis

Blattmann¹,

Milbich²,

Dorkenwald³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

What would be the effect of locally poking a static scene? We present an approach that learns naturallylooking global articulations caused by a local manipulation at a pixel level. Training requires only videos of moving objects but no information of the underlying manipulation of the physical scene. Our generative model learns to infer natural object dynamics as a response to user interaction and learns about the interrelations between different object body regions. Given a static image of an object and a local poking of a pixel, the approach then predicts how the object would deform over time. In contrast to existing work on video prediction, we do not synthesize arbitrary realistic videos but enable local interactive control of the deformation. Our model is not restricted to particular object categories and can transfer dynamics onto novel unseen object instances. Extensive experiments on diverse objects demonstrate the effectiveness of our approach compared to common video prediction frameworks. Project page is available at https://bit.ly/3cxfA2L.

show abstract

Invertible Neural Networks for Understanding Semantics of Invariances of CNN Representations

Rombach

Esser

Blattmann

et al. 2022

Deep Neural Networks and Data for Automated Driving

View full text Add to dashboard Cite

To tackle increasingly complex tasks, it has become an essential ability of neural networks to learn abstract representations. These task-specific representations and, particularly, the invariances they capture turn neural networks into black-box models that lack interpretability. To open such a black box, it is, therefore, crucial to uncover the different semantic concepts a model has learned as well as those that it has learned to be invariant to. We present an approach based on invertible neural networks (INNs) that (i) recovers the task-specific, learned invariances by disentangling the remaining factor of variation in the data and that (ii) invertibly transforms these recovered invariances combined with the model representation into an equally expressive one with accessible semantic concepts. As a consequence, neural network representations become understandable by providing the means to (i) expose their semantic meaning, (ii) semantically modify a representation, and (iii) visualize individual learned semantic concepts and invariances. Our invertible approach significantly extends the abilities to understand black-box models by enabling post hoc interpretations of state-of-the-art networks without compromising their performance. Our implementation is available at https://compvis.github.io/invariances/.

show abstract

Behavior-Driven Synthesis of Human Dynamics

Cited by 12 publications

References 29 publications

iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis

iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis

Understanding Object Dynamics for Interactive Image-to-Video Synthesis

Invertible Neural Networks for Understanding Semantics of Invariances of CNN Representations

Contact Info

Product

Resources

About