Taking visual motion prediction to new heightfields

Ehrhardt, Sébastien; Monszpart, Áron; Mitra, Niloy J.; Vedaldi, Andrea

doi:10.1016/j.cviu.2019.02.005

Cited by 14 publications

(16 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since it is difficult to generate realistic data that includes complex physical interactions, most approaches either rely on short videos with limited exposition to physics [13,1,2] or on synthetically generated data [8,9,12,30]. This problem of lacking good realistic environments with rich physical interactions also governs the research on reinforcement learning [31], where the community often relies on game-like environments [32,33,34,35].…”

Section: Simulations and Machine Learningmentioning

confidence: 99%

Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions

Wagner

Basevi

Shetty

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In-depth scene descriptions and question answering tasks have greatly increased the scope of today's definition of scene understanding. While such tasks are in principle open ended, current formulations primarily focus on describing only the current state of the scenes under consideration. In contrast, in this paper, we focus on the future states of the scenes which are also conditioned on actions. We posit this as a question answering task, where an answer has to be given about a future scene state, given observations of the current scene, and a question that includes a hypothetical action. Our solution is a hybrid model which integrates a physics engine into a question answering architecture in order to anticipate future scene states resulting from object-object interactions caused by an action. We demonstrate first results on this challenging new problem and compare to baselines, where we outperform fully data-driven end-to-end learning approaches.

show abstract

Section: Simulations and Machine Learningmentioning

confidence: 99%

Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions

Wagner

Basevi

Shetty

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…In recent years, researchers have been building differentiable physics simulators in various forms [1], [3], [13], [14]. Our (a) shows the initial states of a Newton's cradle, based on which both the Interaction Networks and Propagation Networks try to predict future states; (b-i) The Interaction Networks can only propagate the force along a single relation at a time step, thus results in a false prediction (c-i); (b-ii) Our proposed method can propagate the force correctly which leads to the correct prediction (c-ii); (d) A downstream task where we aim to achieve a specific goal using the learned model; (e-i) Model-based control methods fail to produce the correct control using Interaction Networks while (e-ii) our model can provide the desired control signal.…”

Section: Related Work a Differentiable Physics Simulatorsmentioning

confidence: 99%

Propagation Networks for Model-Based Control Under Partial Observation

Zhu

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

100

109

View full text Add to dashboard Cite

There has been an increasing interest in learning dynamics simulators for model-based control. Compared with off-the-shelf physics engines, a learnable simulator can quickly adapt to unseen objects, scenes, and tasks. However, existing models like interaction networks only work for fully observable systems; they also only consider pairwise interactions within a single time step, both restricting their use in practical systems. We introduce Propagation Networks (PropNet), a differentiable, learnable dynamics model that handles partially observable scenarios and enables instantaneous propagation of signals beyond pairwise interactions. Experiments show that our propagation networks not only outperform current learnable physics engines in forward simulation, but also achieve superior performance on various control tasks. Compared with existing model-free deep reinforcement learning algorithms, model-based control with propagation networks is more accurate, efficient, and generalizable to new, partially observable scenes and tasks. k∈Ni e k,t ), i = 1 . . . |O|,where o i,t = x i,t , a o i , p i,t denotes object i at time t, u k and v k are the receiver and sender of relation r k , and N i denotes the relations where object i is the receiver. B. Propagation NetworksIN defines a flexible and efficient model for explicit reasoning of objects and their relations in a complex system. It can handle a variable number of objects and relations and has performed well in domains like n-body systems, bouncing

show abstract

“…Another method was proposed to obtain the property vector for each object from the video to predict the future trajectories of objects and infer the interpretable physical values, such as mass or rebound coefficients, using principle component analysis [4]. Some studies were conducted to predict the future movement of balls rolling in elliptical bowls [8,32], and others were conducted to predict future object states and future frames by inputting short video sequences [9,33] using the spatial transform network [34]. A study was also conducted to predict the future trajectories of a bouncing ball [5] by using a variational recurrent neural network [35].…”

Section: Learning Explicit Physicsmentioning

confidence: 99%

Future-Frame Prediction for Fast-Moving Objects with Motion Blur

Dohae

Lee

2020

Sensors

View full text Add to dashboard Cite

We propose a deep neural network model that recognizes the position and velocity of a fast-moving object in a video sequence and predicts the object’s future motion. When filming a fast-moving subject using a regular camera rather than a super-high-speed camera, there is often severe motion blur, making it difficult to recognize the exact location and speed of the object in the video. Additionally, because the fast moving object usually moves rapidly out of the camera’s field of view, the number of captured frames used as input for future-motion predictions should be minimized. Our model can capture a short video sequence of two frames with a high-speed moving object as input, use motion blur as additional information to recognize the position and velocity of the object, and predict the video frame containing the future motion of the object. Experiments show that our model has significantly better performance than existing future-frame prediction models in determining the future position and velocity of an object in two physical scenarios where a fast-moving two-dimensional object appears.

show abstract

Taking visual motion prediction to new heightfields

Cited by 14 publications

References 7 publications

Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions

Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions

Propagation Networks for Model-Based Control Under Partial Observation

Future-Frame Prediction for Fast-Moving Objects with Motion Blur

Contact Info

Product

Resources

About