AI Generality and Spearman’s Law of Diminishing Returns

Abstract


 
 
 Many areas of AI today use benchmarks and competitions with larger and wider sets of tasks. This tries to deter AI systems (and research effort) from specialising to a single task, and encourage them to be prepared to solve previously unseen tasks. It is unclear, however, whether the methods with best performance are actually those that are most general and, in perspective, whether the trend moves towards more general AI systems. This question has a striking similarity with the analysis of the so-called positive manifold and general factors in the area of human intelligence. In this paper, we first show how the existence of a manifold (positive average pairwise task correlation) can also be analysed in AI, and how this relates to the notion of agent generality, from the individual and the populational points of view. From the populational perspective, we analyse the following question: is this manifold correlation higher for the most or for the least able group of agents? We contrast this analysis with one of the most controversial issues in human intelligence research, the so-called Spearman's Law of Diminishing Returns (SLODR), which basically states that the relevance of a general factor diminishes for most able human groups. We perform two empirical studies on these issues in AI. We analyse the results of the 2015 general video game AI (GVGAI) competition, with games as tasks and "controllers" as agents, and the results of a synthetic setting, with modified elementary cellular automata (ECA) rules as tasks and simple interactive programs as agents. In both cases, we see that SLODR doesnot appear. The data, and the use of just two scenarios, does not clearly support the reverse either, a Universal Law of Augmenting Returns (ULOAR), but calls for more experiments on this question.
 
 

Basic functional trade-offs in cognition: An integrative framework

“…The analysis of cognitive trade-offs also draws attention to the issue of similarities and differences between biological and artificial intelligence systems with regard to trade-off architectures (e.g., Hernández-Orallo, 2016). For example, advanced neural networks optimized for accuracy and/or speed in object recognition can be surprisingly fragile against extremely small perturbations-sometimes consisting of a single pixel-that have been specifically engineered to "fool" their algorithms, but may be invisible to human observers (Akhtar & Mian, 2018;Moosavi-Dezfooli et al, 2017).…”

The Archimedean trap: Why traditional reinforcement learning will probably not yield AGI

“…x 1 • • • exhibiting diminishing returns, in the sense that the measurements of x i and x j are approximately equal for all large enough i, j. If human intelligence is non-Archimedean, this could potentially shed light on a psychometrical phenomenon called Spearman's Law of Diminishing Returns (Spearman, 1927;Blum and Holling, 2017;Hernández-Orallo, 2019), the empirical tendency of cognitive ability tests to be less correlated in high-intelligence populations. Even tiny measurement errors would eventually dominate the test result differences as the true measurements plateau.…”

The Archimedean trap: Why traditional reinforcement learning will probably not yield AGI

“…This suggests a general law of diminishing returns: any time a non-Archimedean significantly-ordered structure (X, ≪) is measured using real numbers, if the measurement does not blatantly violate ≪ (in other words, if there are no x ≪ y such that x is given a larger real-number measurement than y), then there will inevitably be elements x 0 ≪ x 1 ≪ • • • exhibiting diminishing returns, in the sense that the measurements of x i and x j are approximately equal for large i, j. If human intelligence is non-Archimedean, this could potentially shed light on a psychometrical phenomenon called Spearman's Law of Diminishing Returns [29] [8] [13], the empirical tendency of cognitive ability tests to be less correlated in high-intelligence populations. Even tiny measurement errors would eventually dominate the test result differences as the true results plateau.…”