Disentangling What and Where for 3D Object-Centric Representations Through Active Inference

Maele, Toon Van de; Verbelen, Tim; Çatal, Ozan; Dhoedt, Bart

doi:10.48550/arxiv.2108.11762

“…In [14], the authors propose a column-like structure, similar to hypercolumns typical of the human neocortex. In [53], the authors build upon cortical columns implemented as separate neural networks called Cortical Column Networks (CCN). Their framework aims at representing part-whole relationships in scenes to learn object-centric representations for classification.…”

Section: Related Workmentioning

confidence: 99%

Interpretable part-whole hierarchies and conceptual-semantic relationships in neural networks

Garau¹,

Bisagno²,

Sambugaro³

et al. 2022

Preprint

0

View full text Add to dashboard Cite

Figure 1. [Better seen in color]. Overview of the proposed solution. Our Agglomerator is a novel architecture for vision applications, in which column structure (c) mimics hyper-columns typical of the human visual cortex [14]. The input data (a) is fed to the columns using a patch-based embedding (b). The Agglomerator architecture iteratively routes the information across its structure, creating a neural representation of each image, similar to neural fields [37]. In the neural representation, part-whole hierarchies (d) emerge at different levels of the columns. The same column can represent the same patch of the image with different levels of abstraction (e.g., the ears, the head, and the dog) corresponding to each level in the column. Neighbor columns agree on a part representation (e.g ears, head) at lower levels, ideally representing the same whole (e.g. dog) at the top level. The resulting feature space represents the conceptual-semantic relationships between data (e) resembling the human hierarchical organization [38]. Samples belonging to the same super-class (e.g., animals, vehicles) are clustered together, with conceptually close categories (e.g., birds and airplanes) represented on the edge of the super-classes.

show abstract

“…Recently, deep learning techniques were proposed to learn such generative models from high dimensional sensor data [33,7,27], which paves the way to more complex application areas such as robot perception [14]. In particular, Van de Maele et al [16,18] introduced object-centric, deep active inference models that enable an agent to infer the pose and identity of a particular object instance. However, this model was restricted to identify unique object instances, i.e.…”

Section: Introductionmentioning

confidence: 99%

Disentangling Shape and Pose for Object-Centric Deep Active Inference Models

Ferraro

¹

,

Maele

²

,

Mazzaglia

³

et al. 2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

Active inference is a first principles approach for understanding the brain in particular, and sentient agents in general, with the single imperative of minimizing free energy. As such, it provides a computational account for modelling artificial intelligent agents, by defining the agent's generative model and inferring the model parameters, actions and hidden state beliefs. However, the exact specification of the generative model and the hidden state space structure is left to the experimenter, whose design choices influence the resulting behaviour of the agent. Recently, deep learning methods have been proposed to learn a hidden state space structure purely from data, alleviating the experimenter from this tedious design task, but resulting in an entangled, non-interpreteable state space. In this paper, we hypothesize that such a learnt, entangled state space does not necessarily yield the best model in terms of free energy, and that enforcing different factors in the state space can yield a lower model complexity. In particular, we consider the problem of 3D object representation, and focus on different instances of the ShapeNet dataset. We propose a model that factorizes object shape, pose and category, while still learning a representation for each factor using a deep neural network. We show that models, with best disentanglement properties, perform best when adopted by an active agent in reaching preferred observations.

show abstract

“…Recently, deep learning techniques were proposed to learn such generative models from high dimensional sensor data [33,7,27], which paves the way to more complex application areas such as robot perception [14]. In particular, Van de Maele et al [16,18] introduced object-centric, deep active inference models that enable an agent to infer the pose and identity of a particular object instance. However, this model was restricted to identify unique object instances, i.e.…”

Section: Introductionmentioning

confidence: 99%

Disentangling Shape and Pose for Object-Centric Deep Active Inference Models

Ferraro¹,

Maele²,

Mazzaglia³

et al. 2022

Preprint

Self Cite

0

View full text Add to dashboard Cite

Active inference is a first principles approach for understanding the brain in particular, and sentient agents in general, with the single imperative of minimizing free energy. As such, it provides a computational account for modelling artificial intelligent agents, by defining the agent's generative model and inferring the model parameters, actions and hidden state beliefs. However, the exact specification of the generative model and the hidden state space structure is left to the experimenter, whose design choices influence the resulting behaviour of the agent. Recently, deep learning methods have been proposed to learn a hidden state space structure purely from data, alleviating the experimenter from this tedious design task, but resulting in an entangled, non-interpreteable state space. In this paper, we hypothesize that such a learnt, entangled state space does not necessarily yield the best model in terms of free energy, and that enforcing different factors in the state space can yield a lower model complexity. In particular, we consider the problem of 3D object representation, and focus on different instances of the ShapeNet dataset. We propose a model that factorizes object shape, pose and category, while still learning a representation for each factor using a deep neural network. We show that models, with best disentanglement properties, perform best when adopted by an active agent in reaching preferred observations.

show abstract

Disentangling What and Where for 3D Object-Centric Representations Through Active Inference

Cited by 3 publications

References 3 publications

Interpretable part-whole hierarchies and conceptual-semantic relationships in neural networks

Interpretable part-whole hierarchies and conceptual-semantic relationships in neural networks

Disentangling Shape and Pose for Object-Centric Deep Active Inference Models

Disentangling Shape and Pose for Object-Centric Deep Active Inference Models

Contact Info

Product

Resources

About