Self-Modification and Mortality in Artificial Agents

Orseau, Laurent; Ring, Mark

doi:10.1007/978-3-642-22887-2_1

Cited by 31 publications

(39 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…That is, UAI has the potential to arrive at definite answers to various questions regarding the social behavior of super-intelligences. Some formalizations and semi-formal answers have recently appeared in the award-winning papers [OR11,RO11].…”

Section: Social Questionsmentioning

confidence: 99%

One Decade of Universal Artificial Intelligence

Hutter¹

2012

Atlantis Thinking Machines

View full text Add to dashboard Cite

The first decade of this century has seen the nascency of the first mathematical theory of general artificial intelligence. This theory of Universal Artificial Intelligence (UAI) has made significant contributions to many theoretical, philosophical, and practical AI questions. In a series of papers culminating in book (Hutter, 2005), an exciting sound and complete mathematical model for a super intelligent agent (AIXI) has been developed and rigorously analyzed. While nowadays most AI researchers avoid discussing intelligence, the awardwinning PhD thesis (Legg, 2008) provided the philosophical embedding and investigated the UAI-based universal measure of rational intelligence, which is formal, objective and non-anthropocentric. Recently, effective approximations of AIXI have been derived and experimentally investigated in JAIR paper ). This practical breakthrough has resulted in some impressive applications, finally muting earlier critique that UAI is only a theory. For the first time, without providing any domain knowledge, the same agent is able to self-adapt to a diverse range of interactive environments. For instance, AIXI is able to learn from scratch to play TicTacToe, Pacman, Kuhn Poker, and other games by trial and error, without even providing the rules of the games.These achievements give new hope that the grand goal of Artificial General Intelligence is not elusive.This article provides an informal overview of UAI in context. It attempts to gently introduce a very theoretical, formal, and mathematical subject, and discusses philosophical and technical ingredients, traits of intelligence, some social questions, and the past and future of UAI.

show abstract

Section: Social Questionsmentioning

confidence: 99%

One Decade of Universal Artificial Intelligence

Hutter¹

2012

Atlantis Thinking Machines

View full text Add to dashboard Cite

show abstract

“…We (very) briefly summarize the definition of a universal agent, based on AIXI [3,4], following Orseau & Ring [8,13].…”

Section: Notation and Agent Frameworkmentioning

confidence: 99%

“…A knowledge-seeking agent (KSA) [8,13,11], chooses actions to maximize its knowledge of the environment (by reducing ρ(o 1:t | a 1:t ) through elimination of inconsistent environments) as quickly as possible; thus its utility function is u(ao 1:t ) = −ρ(o 1:t | a 1:t ). A prediction-seeking agent (PSA) [8,13] tries to maximize the accuracy of its predictions: u(ao 1:t ) = 1 if o t = arg max o ρ(o ≺t o | a 1:t ), and 0 otherwise.…”

Section: Notation and Agent Frameworkmentioning

confidence: 99%

“…In this and previous work we have begun developing formal models in which these assumptions are relaxed and in which the AGI agent is forced, bit by bit, to inhabit the same universe that we do. In our previous work, for example, we considered the theoretical consequences of taking various universal intelligent agents such as AIXI [3] and embedding their source code into their environment such that it can be modified by the agents themselves [8] or even by the environment [13], as is the case in the real world.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Memory Issues of Intelligent Agents

Orseau

Ring

2012

Artificial General Intelligence

Self Cite

View full text Add to dashboard Cite

Abstract. Theoretical models of artificial general intelligence, such as AIXI [3], typically consider an intelligent agent to have unlimited computational resources, allowing it to keep a perfect memory of its entire interaction history with its environment. In the real world, an agent's memory is part of the environment, which means that the latter can modify the former. This paper develops a theoretical framework for examining the implications of such real-world memory on universal intelligent agents. Within this framework we are able to show, for example, that in certain environments optimality can be achieved only with truly stochastic behaviors, and that guarantees about the trustworthiness of memories are difficult to obtain even with infinite computational power. To describe the probability of an agent's memory state, we propose an adaptation of the universal prior for the passive and the active case.

show abstract

“…The AGI-11 conference included three papers (Orseau and Ring, 2011a; Ring and Orseau, 2011b;Dewey, 2011) that employed the mathematics of rational agents to analyze ways that AI agents may fail to satisfy the intentions of their designers. Omohundro (2008) and Bostrom (forthcoming) described secondary AI motivations that are implied by a wide variety of primary motivations and that may drive unintended behaviors threatening humans.…”

Section: Introductionmentioning

confidence: 99%

Avoiding Unintended AI Behaviors

Hibbard

2012

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Artificial intelligence (AI) systems too complex for predefined environment models and actions will need to learn environment models and to choose actions that optimize some criteria. Several authors have described mechanisms by which such complex systems may behave in ways not intended in their designs. This paper describes ways to avoid such unintended behavior. For hypothesized powerful AI systems that may pose a threat to humans, this paper proposes a two-stage agent architecture that avoids some known types of unintended behavior. For the first stage of the architecture this paper shows that the most probable finite stochastic program to model a finite history is finitely computable, and that there is an agent that makes such a computation without any unintended instrumental actions.

show abstract

Self-Modification and Mortality in Artificial Agents

Cited by 31 publications

References 10 publications

One Decade of Universal Artificial Intelligence

One Decade of Universal Artificial Intelligence

Memory Issues of Intelligent Agents

Avoiding Unintended AI Behaviors

Contact Info

Product

Resources

About