Belief Regulated Dual Propagation Nets for Learning Action Effects on Groups of Articulated Objects

Abstract

In recent years, graph neural networks have been successfully applied for learning the dynamics of complex and partially observable physical systems. However, their use in the robotics domain is, to date, still limited. In this paper, we introduce Belief Regulated Dual Propagation Networks (BRDPN), a general purpose learnable physics engine, which enables a robot to predict the effects of its actions in scenes containing groups of articulated multi-part objects. Specifically, our framework extends the recently proposed propagation networks (PropNets) and consists of two complementary components, a physics predictor and a belief regulator. While the former predicts the future states of the object(s) manipulated by the robot, the latter constantly corrects the robots knowledge regarding the objects and their relations. Our results showed that after trained in a simulator, the robot could reliably predict the consequences of its actions in object trajectory level and exploit its own interaction experience to correct its belief about the state of the world, enabling better predictions in partially observable environments. Furthermore, the trained model was transferred to the real world and its capabilities were verified in correctly predicting trajectories of pushed interacting objects whose joint relations were initially unknown. We compared our BRDPN against the original PropNets and showed that BRDPN can perform consistently well even if the relations between the objects are not explicitly given but instead predicted from observations.

Object and Relation Centric Representations for Push Effect Prediction

“…Furthermore, the training of the network has been improved by the use of scheduled sampling [31] and greater data distribution. These novel contributions decrease the errors for long-horizon prediction tasks, and in Section V-E, our new results have been shown to surpass the ones in [30]. More specifically, the general contributions of our framework can be listed as follows:…”

“…An early version of this work was published in [30]. However, this paper significantly extend the work in several important directions.…”

“…In this section, our framework is evaluated with a realworld dataset, presented in [30]. In this dataset, a UR10 robot arm holds a hammer and use it for pushing objects.…”

“…However, this paper significantly extend the work in several important directions. In [30], for physics prediction and parameter estimation, two independents networks were required. By employing a new weight-sharing mechanism that allows these tasks to share object-and relation-centric representations, the number of learnable parameters is decreased by about thirty percent.…”

Graph-based Task-specific Prediction Models for Interactions between Deformable and Rigid Objects

“…A new trend in action effect prediction employs graph neural networks to learn system dynamics with the ability to generalize to scenarios with a different number of objects [3]. Graph neural networks have been also used to predict the motion of stacked block towers [21], the effects of pushing into a scene of rigid objects [22], and of interacting with a connected set of rigid objects [23].…”