The Archimedean trap: Why traditional reinforcement learning will probably not yield AGI

2023

Preprint

We bring together foundational theories of meaning and a mathematical formalism of artificial general intelligence to provide a mechanistic explanation of meaning, communication and symbol emergence. We establish circumstances under which a machine might mean what we think it means by what it says, or comprehend what we mean by what we say. We conclude that a language model such as ChatGPT does not comprehend or engage in meaningful communication with humans, though it may exhibit complex behaviours such as theory of mind.

Section: Extending the Formalismmentioning

confidence: 99%

On the Computation of Meaning, Language Models and Incomprehensible Horrors

2023

Preprint

“…However, to simulate feelings that result in human-like behaviour is a more difficult proposition. Rather than trying to describe human-like feelings, we simplify our analysis by assuming the preferences [20] n o which are determined by experience of feelings.…”

Section: Extending the Formalismmentioning

confidence: 99%

On the Computation of Meaning, Language Models and Incomprehensible Horrors

2023

Preprint

[Accepted to the 16th Annual Conference on Artificial General Intelligence] We bring together foundational theories of meaning and a mathematical formalism of artificial general intelligence to provide a mechanistic explanation of meaning, communication and symbol emergence. We establish circumstances under which a machine might mean what we think it means by what it says, or comprehend what we mean by what we say. We conclude that a language model such as ChatGPT does not comprehend or engage in meaningful communication with humans, though it may exhibit complex behaviours such as theory of mind.

“…These urges impose an order of preference on decisions, and a threshold beyond which the utility is "good enough" to indicate completion [18,27]. This ordering is akin to preference based reinforcement learning [39], which allows adaptation to "non-Archimedean tasks whose rewards are too rich to express using real numbers" [40]. Given preference, decisions can be evaluated and ostensive definitions constructed.…”

Section: Ambiguity and Correctnessmentioning

confidence: 99%

Computable Artificial General Intelligence

2022

Preprint

The general reinforcement learning agent AIXI may be the only mathematical formalism of artificial general intelligence (AGI) supported by proof that its performance is optimal. This is achieved using compression as a proxy for intelligence. Unfortunately, AIXI is incomputable and claims of its optimality were later shown to be subjective. This paper proposes an alternative, supported by proof, which overcomes both problems. Integrating research from cognitive science (enactivism), philosophy (intension and extension), machine learning and planning (satplan), an arbitrary task is given mathematical rigour. This serves as an enactive model of learning and reasoning within which a description of intelligence is formalised. Instead of compression we use weakness as our proxy, and the result is both computable and objective. We formally prove that maximising weakness maximises intelligence. This proof is then further supported with experimental results comparing weakness and description length (the closest analogue to compression possible under the enactive model that would not reintroduce the problem of subjective performance). Our results show that weakness outperforms description length, and is a better proxy for intelligence. The foremost limitation is that intelligence as we have defined it is computationally complex, but may be coupled with domain specific inductive biases to make real-world domains of practical significance tractable (e.g. the domain of all tasks a human would undertake). Like AIXI this is not intended to be a panacea but to demonstrate useful principles, which may be integrated with existing tools such as neural networks to improve performance.