On the Binding Problem in Artificial Neural Networks

Abstract

Contemporary neural networks still fall short of human-level generalization, which extends far beyond our direct experiences. In this paper, we argue that the underlying cause for this shortcoming is their inability to dynamically and flexibly bind information that is distributed throughout the network. This binding problem affects their capacity to acquire a compositional understanding of the world in terms of symbol-like entities (like objects), which is crucial for generalizing in predictable and systematic ways. To address this issue, we propose a unifying framework that revolves around forming meaningful entities from unstructured sensory inputs (segregation), maintaining this separation of information at a representational level (representation), and using these entities to construct new inferences, predictions, and behaviors (composition). Our analysis draws inspiration from a wealth of research in neuroscience and cognitive psychology, and surveys relevant mechanisms from the machine learning literature, to help identify a combination of inductive biases that allow symbolic information processing to emerge naturally in neural networks. We believe that a compositional approach to AI, in terms of grounded symbol-like representations, is of fundamental importance for realizing human-level generalization, and we hope that this paper may contribute towards that goal as a reference and inspiration.

The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

“…Systematic generalization (Fodor et al, 1988) is a desired property for neural networks to extrapolate compositional rules seen during training beyond training distribution: for example, performing different combinations of known rules or applying them to longer problems. Despite the progress of artificial neural networks in recent years, the problem of systematic generalization still remains unsolved (Fodor and McLaughlin, 1990;Lake and Baroni, 2018;Liska et al, 2018;Greff et al, 2020;Hupkes et al, 2020). While there has been much progress in the past years (Bahdanau et al, 2019;Korrel et al, 2019;Lake, 2019;Li et al, 2019;Russin et al, 2019), in particular on the popular SCAN dataset (Lake and where some methods even achieve 100% accuracy by introducing some non-trivial symbolic components into the system Liu et al, 2020), the flexibility of such solutions is questionable.…”

Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition

“…Object-centric models ("Slots" and "Capsules"): Objects are how people interact with the world and are therefore central to human scene understanding [Scholl, 2001, Spelke, 1990. Visual objects are formed by (bottom-up) part-whole matching and Gestalt processes interacting with (top-down) objectness priors and knowledge of object categories [Greff et al, 2020, Vecera, 2000, Wagemans et al, 2012. Object-centric models use these processes to discover objects and segregate their representations into different "slots" [Greff et al, 2020, Goyal et al, 2019.…”

“…Visual objects are formed by (bottom-up) part-whole matching and Gestalt processes interacting with (top-down) objectness priors and knowledge of object categories [Greff et al, 2020, Vecera, 2000, Wagemans et al, 2012. Object-centric models use these processes to discover objects and segregate their representations into different "slots" [Greff et al, 2020, Goyal et al, 2019. Attention mechanisms have played a major role in object-centric models by enabling the iterative discovery and representation of an object's properties.…”

Conditional Object-Centric Learning from Video

“…and is crucial for generalizing in predictable and systematic ways. Object-centric representations have the potential to greatly improve sample efficiency, robustness, generalization to new tasks, and interpretability of machine learning algorithms (Greff et al, 2020). In this work, we focus on the aspect of modeling motion of objects from video, because of its synergistic relationship with object-centric representations: On the one hand, objects support learning an efficient dynamics model by factorizing the scene into approximately independent parts with only sparse interactions.…”