Human-like systematic generalization through a meta-learning neural network

Abstract

The power of human language and thought arises from systematic compositionality—the algebraic ability to understand and produce novel combinations from known components. Fodor and Pylyshyn1 famously argued that artificial neural networks lack this capacity and are therefore not viable models of the mind. Neural networks have advanced considerably in the years since, yet the systematicity challenge persists. Here we successfully address Fodor and Pylyshyn’s challenge by providing evidence that neural networks can achieve human-like systematicity when optimized for their compositional skills. To do so, we introduce the meta-learning for compositionality (MLC) approach for guiding training through a dynamic stream of compositional tasks. To compare humans and machines, we conducted human behavioural experiments using an instruction learning paradigm. After considering seven different models, we found that, in contrast to perfectly systematic but rigid probabilistic symbolic models, and perfectly flexible but unsystematic neural networks, only MLC achieves both the systematicity and flexibility needed for human-like generalization. MLC also advances the compositional skills of machine learning systems in several systematic generalization benchmarks. Our results show how a standard neural network architecture, optimized for its compositional skills, can mimic human systematic generalization in a head-to-head comparison.

A mathematical theory of relational generalization in transitive inference

“…the ability to conceptualize prior experience in terms of components that can be re-configured in a novel situation [167,168]. More broadly, compositional generalization has long been understood to be a crucial component for human-like learning and generalization [163], in large part due to the diversity and breadth of its applications: compositional generalization often involves the composition of many rules, relations, or attributes [163,[169][170][171]. The comparative simplicity of TI enabled us to identify how minimally structured learning systems can implement the inductive biases needed for this task.…”

Clothing image attribute editing based on generative adversarial network, with reference to an upper garment

“…Although these models are large in scale and contain billions of parameters, it’s undeniable that LLMs have demonstrated impressive capabilities in image generation tasks. Recent study result (Lake and Baroni, 2023) shows that neural networks can achieve human-like systematicity when optimized for their compositional skills. In addition to LLMs, a new visual model has also attracted attention and it is the Large Vision Model (LVM).…”

Cross-Domain Knowledge Transfer without Retraining to Facilitating Seamless Knowledge Application in Large Language Models

“…Meta-learning was shown to enhance the flexibility of AI systems by enabling them to learn new tasks quickly with minimal data input [23], [58]- [60]. Algorithms that adjust their learning rules based on the task at hand demonstrated high versatility and adaptability [60]- [62].…”

“…Research into meta-learner architectures provided insights into optimizing model parameters for rapid adaptability [63], [67], [68]. The use of metalearning in complex scenarios involving multiple tasks and domains was explored, showing potential for scalable, efficient learning systems [36], [58], [69]- [71]. Findings also indicated that meta-learning could reduce the computational costs associated with adapting models to new environments [60], [62], [72].…”