Active Vision for Robot Manipulators Using the Free Energy Principle

Maele, Toon Van de; Verbelen, Tim; Çatal, Ozan; Boom, Cedric De; Dhoedt, Bart

doi:10.3389/fnbot.2021.642780

Cited by 22 publications

(20 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar generative models can be used for learning machine vision using pixel observations (Eslami et al, 2018 ; Van de Maele et al, 2021b ). In this case, the system is trained to make inferences about the scene s , given images o t and corresponding absolute viewpoints v t .…”

Section: Methodsmentioning

confidence: 99%

Embodied Object Representation Learning and Recognition

et al. 2022

Self Cite

View full text Add to dashboard Cite

Scene understanding and decomposition is a crucial challenge for intelligent systems, whether it is for object manipulation, navigation, or any other task. Although current machine and deep learning approaches for object detection and classification obtain high accuracy, they typically do not leverage interaction with the world and are limited to a set of objects seen during training. Humans on the other hand learn to recognize and classify different objects by actively engaging with them on first encounter. Moreover, recent theories in neuroscience suggest that cortical columns in the neocortex play an important role in this process, by building predictive models about objects in their reference frame. In this article, we present an enactive embodied agent that implements such a generative model for object interaction. For each object category, our system instantiates a deep neural network, called Cortical Column Network (CCN), that represents the object in its own reference frame by learning a generative model that predicts the expected transform in pixel space, given an action. The model parameters are optimized through the active inference paradigm, i.e., the minimization of variational free energy. When provided with a visual observation, an ensemble of CCNs each vote on their belief of observing that specific object category, yielding a potential object classification. In case the likelihood on the selected category is too low, the object is detected as an unknown category, and the agent has the ability to instantiate a novel CCN for this category. We validate our system in an simulated environment, where it needs to learn to discern multiple objects from the YCB dataset. We show that classification accuracy improves as an embodied agent can gather more evidence, and that it is able to learn about novel, previously unseen objects. Finally, we show that an agent driven through active inference can choose their actions to reach a preferred observation.

show abstract

Section: Methodsmentioning

confidence: 99%

Embodied Object Representation Learning and Recognition

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…rotation and translation), moving the camera viewpoint. This setup closely mimicks a robot manipulator with an in-hand camera, but without kinematic constraints [26].…”

Section: Methodsmentioning

confidence: 99%

“…However, recent work also focused on active vision. In [26] a generative model learning representations of a whole 3D scene was used for an active inference agent, whereas in [2] an explicit what and where stream were modeled for classifying MNIST digits.…”

Section: Related Workmentioning

confidence: 99%

Disentangling What and Where for 3D Object-Centric Representations Through Active Inference

Maele

Verbelen

Çatal

et al. 2021

Communications in Computer and Information Science

Self Cite

View full text Add to dashboard Cite

Although modern object detection and classification models achieve high accuracy, these are typically constrained in advance on a fixed train set and are therefore not flexible to deal with novel, unseen object categories. Moreover, these models most often operate on a single frame, which may yield incorrect classifications in case of ambiguous viewpoints. In this paper, we propose an active inference agent that actively gathers evidence for object classifications, and can learn novel object categories over time. Drawing inspiration from the human brain, we build object-centric generative models composed of two information streams, a what-and a where-stream. The what-stream predicts whether the observed object belongs to a specific category, while the where-stream is responsible for representing the object in its internal 3D reference frame. We show that our agent (i) is able to learn representations for many object categories in an unsupervised way, (ii) achieves state-of-theart classification accuracies, actively resolving ambiguity when required and (iii) identifies novel object categories. Furthermore, we validate our system in an end-to-end fashion where the agent is able to search for an object at a given pose from a pixel-based rendering. We believe that this is a first step towards building modular, intelligent systems that can be used for a wide range of tasks involving three dimensional objects.

show abstract

“…In the active inference literature, multivariate Gaussian (also known as normal) distributions with a diagonal covariance matrix have been largely adopted since the initial works on VAEs [ 32 , 33 , 68 , 86 ]. Similarly, numerous latent state space models have adopted a Gaussian structure of their latent space [ 48 , 49 , 87 , 88 ], but also more complex mixture models have been proposed [ 50 ].…”

Section: Variational World Modelsmentioning

confidence: 99%

The Free Energy Principle for Perception and Action: A Deep Learning Perspective

Mazzaglia

Verbelen

Çatal

et al. 2022

Entropy

Self Cite

View full text Add to dashboard Cite

The free energy principle, and its corollary active inference, constitute a bio-inspired theory that assumes biological agents act to remain in a restricted set of preferred states of the world, i.e., they minimize their free energy. Under this principle, biological agents learn a generative model of the world and plan actions in the future that will maintain the agent in an homeostatic state that satisfies its preferences. This framework lends itself to being realized in silico, as it comprehends important aspects that make it computationally affordable, such as variational inference and amortized planning. In this work, we investigate the tool of deep learning to design and realize artificial agents based on active inference, presenting a deep-learning oriented presentation of the free energy principle, surveying works that are relevant in both machine learning and active inference areas, and discussing the design choices that are involved in the implementation process. This manuscript probes newer perspectives for the active inference framework, grounding its theoretical aspects into more pragmatic affairs, offering a practical guide to active inference newcomers and a starting point for deep learning practitioners that would like to investigate implementations of the free energy principle.

show abstract

Active Vision for Robot Manipulators Using the Free Energy Principle

Cited by 22 publications

References 42 publications

Embodied Object Representation Learning and Recognition

Embodied Object Representation Learning and Recognition

Disentangling What and Where for 3D Object-Centric Representations Through Active Inference

The Free Energy Principle for Perception and Action: A Deep Learning Perspective

Contact Info

Product

Resources

About