Extracting Money from Causal Decision Theorists

Oesterheld, Caspar; Conitzer, Vincent

doi:10.1093/pq/pqaa086

Cited by 5 publications

(4 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, there exists no agent that complies with standard BDT in this problem. Compare the example of Oesterheld and Conitzer [18] and Spencer [28]; also see Demski and Garrabrant ([7], Sect. 2.1) for a discussion of another, subtler issue that arises from logical omniscience and introspection.…”

Section: Computational Constraints and Paradoxes Of Self-referencementioning

confidence: 99%

“…Paradoxes of self-reference, strategic interactions, and counterfactuals A second problem with BDT and logical omniscience more generally is that it creates inconsistencies if the values of different available options depend on what the agent chooses. As an example, consider the following decision problem, which we will call the Simplified Adversarial Offer (SAO) (after a decision problem introduced by [18]). Imagine that an artificial agent chooses between two available alternatives a 0 and a 1 , where a 0 is known to pay off 1 /2 with certainty, and a 1 is known to pay off 1 if the agent's program run on this decision problem chooses a 0 , and 0 otherwise.…”

Section: Computational Constraints and Paradoxes Of Self-referencementioning

confidence: 99%

See 1 more Smart Citation

A Theory of Bounded Inductive Rationality

Oesterheld

Demski

Conitzer

2023

Electron. Proc. Theor. Comput. Sci.

Self Cite

View full text Add to dashboard Cite

The dominant theories of rational choice assume logical omniscience. That is, they assume that when facing a decision problem, an agent can perform all relevant computations and determine the truth value of all relevant logical/mathematical claims. This assumption is unrealistic when, for example, we offer bets on remote digits of π or when an agent faces a computationally intractable planning problem. Furthermore, the assumption of logical omniscience creates contradictions in cases where the environment can contain descriptions of the agent itself. Importantly, strategic interactions as studied in game theory are decision problems in which a rational agent is predicted by its environment (the other players). In this paper, we develop a theory of rational decision making that does not assume logical omniscience. We consider agents who repeatedly face decision problems (including ones like betting on digits of π or games against other agents). The main contribution of this paper is to provide a sensible theory of rationality for such agents. Roughly, we require that a boundedly rational inductive agent tests each efficiently computable hypothesis infinitely often and follows those hypotheses that keep their promises of high rewards. We then prove that agents that are rational in this sense have other desirable properties. For example, they learn to value random and pseudo-random lotteries at their expected reward. Finally, we consider strategic interactions between different agents and prove a folk theorem for what strategies bounded rational inductive agents can converge to.

show abstract

Section: Computational Constraints and Paradoxes Of Self-referencementioning

confidence: 99%

Section: Computational Constraints and Paradoxes Of Self-referencementioning

confidence: 99%

A Theory of Bounded Inductive Rationality

Oesterheld

Demski

Conitzer

2023

Electron. Proc. Theor. Comput. Sci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, Bell et al (2021) show that softmax Q-learners behave more like CDT agents. Albert and Heiner (2001), Mayer, Feldmaier, and Shen (2016), Oesterheld, Demski, and Conitzer (2021), and describe methods of learning that result in EDT-like behavior (see also Oesterheld 2021). We hope that future work in this area will shed light on the feasibility of building learning agents that cooperate against near-copies in real-world, complex, asymmetric scenarios.…”

Section: Cooperation Between Copiesmentioning

confidence: 99%

Foundations of Cooperative AI

Conitzer

Oesterheld

2023

AAAI

View full text Add to dashboard Cite

AI systems can interact in unexpected ways, sometimes with disastrous consequences. As AI gets to control more of our world, these interactions will become more common and have higher stakes. As AI becomes more advanced, these interactions will become more sophisticated, and game theory will provide the tools for analyzing these interactions. However, AI agents are in some ways unlike the agents traditionally studied in game theory, introducing new challenges as well as opportunities. We propose a research agenda to develop the game theory of highly advanced AI agents, with a focus on achieving cooperation.

show abstract

“…Another unique type of failure mode for CDT can be referred to as the money pump dilemma [48]. This involves an agent voluntarily playing a game against a predictor, Omega, who has access to their source code.…”

Section: Predictormentioning

confidence: 99%

Achilles Heels for AGI/ASI via Decision Theoretic Adversaries

Casper¹

2020

Preprint

View full text Add to dashboard Cite

As progress in AI continues to advance at a rapid pace, it is crucial to know how advanced systems will make choices and in what ways they may fail. Machines can already outsmart humans in some domains, and understanding how to safely build systems which may have capabilities at or above the human level is of particular concern. One might suspect that superhumanly-intelligent systems should be modeled as as something which humans, by definition, can't outsmart. However, as a challenge to this assumption, this paper presents the Achilles Heel hypothesis which states that highly-effective goal-oriented systemseven ones that are potentially superintelligent-may nonetheless have stable decision theoretic delusions which cause them to make obviously irrational decisions in adversarial settings. In a survey of relevant dilemmas and paradoxes from the decision theory literature, a number of these potential Achilles Heels are discussed in context of this hypothesis. Several novel contributions are made involving the ways in which these weaknesses could be implanted into a system.

show abstract

Extracting Money from Causal Decision Theorists

Cited by 5 publications

References 26 publications

A Theory of Bounded Inductive Rationality

A Theory of Bounded Inductive Rationality

Foundations of Cooperative AI

Achilles Heels for AGI/ASI via Decision Theoretic Adversaries

Contact Info

Product

Resources

About