General intelligence requires rethinking exploration

Abstract

We are at the cusp of a transition from ‘learning from data’ to ‘learning what data to learn from’ as a central focus of artificial intelligence (AI) research. While the first-order learning problem is not completely solved, large models under unified architectures, such as transformers, have shifted the learning bottleneck from how to effectively train models to how to effectively acquire and use task-relevant data. This problem, which we frame as exploration , is a universal aspect of learning in open-ended domains like the real world. Although the study of exploration in AI is largely limited to the field of reinforcement learning, we argue that exploration is essential to all learning systems, including supervised learning. We propose the problem of generalized exploration to conceptually unify exploration-driven learning between supervised learning and reinforcement learning, allowing us to highlight key similarities across learning settings and open research challenges. Importantly, generalized exploration is a necessary objective for maintaining open-ended learning processes, which in continually learning to discover and solve new problems, provides a promising path to more general intelligence.

Enhancing Algorithm Selection through Comprehensive Performance Evaluation: Statistical Analysis of Stochastic Algorithms

“…In the context of this scenario, the symbol Y j denotes the average value observed in the jth treatment or group, as defined in Formula (7). Similarly, Y ij represents the overall average value across all treatments, as expressed in Formula (8).…”

“…However, securing a high-quality benchmark remains a formidable challenge. The preliminary phase of the benchmarking theory involves delineating the problem domain, an intricate task due to the requisite "uniform" distribution of test functions across the entire expanse of potential functions within the problem domain [7].…”

Generalization through Diversity: Improving Unsupervised Environment Design

“…PLR has been empirically shown to be more scalable and also is able to achieve more robust and better "out-of-distribution" performance than PAIRED. In [Jiang et al, 2021a], authors combined randomized generator and replay mechanism together and proposed PLR ⊥ . Empirically, PLR ⊥ achieves the state-of-the-art in literature (as a method that demands no human expertise), and thus we will adopt PLR ⊥ as our primary baseline in this paper.…”

“…The next set of approaches related to PAIRED [Dennis et al, 2020;Jiang et al, 2021a] rely on regret, which is defined approximately as the difference between the maximum and the mean return of students' policy, to generate new levels:…”

“…To achieve robust and generalizing policies, agents must be exposed to different types of environments that assist in learning different challenges [Cobbe et al, 2020]. To that end, existing research has proposed Unsupervised Environment Design (UED) that can create a curriculum (or distribution) of training scenarios that are adaptive with respect to agent policy [Mehta et al, 2020;Portelas et al, 2020;Wang et al, 2019a;Dennis et al, 2020;Jiang et al, 2021b;Jiang et al, 2021a]. Following the terminology in existing works, we will refer to a particular environment configuration as a level, e.g., an arrangement of blocks in the maze, shape/height of the obstacles in front of the agent, the geometric configuration of racing tracks, etc.…”