An AGI Modifying Its Utility Function in Violation of the Strong Orthogonality Thesis

Abstract

An artificial general intelligence (AGI) might have an instrumental drive to modify its utility function to improve its ability to cooperate, bargain, promise, threaten, and resist and engage in blackmail. Such an AGI would necessarily have a utility function that was at least partially observable and that was influenced by how other agents chose to interact with it. This instrumental drive would conflict with the strong orthogonality thesis since the modifications would be influenced by the AGI’s intelligence. AGIs in highly competitive environments might converge to having nearly the same utility function, one optimized to favorably influencing other agents through game theory. Nothing in our analysis weakens arguments concerning the risks of AGI.

Provably Safe Artificial General Intelligence via Interactive Proofs

“…History is replete with examples of 'power corrupts and absolute power corrupts absolutely' as well as singleton coercive government going wrong. The methods presented here are envisioned in an AGI society with specialized labor and voluntary, negotiated interactions that include permission to alter societal shared values that impact individual utility functions in order to prevent utility function changes from threatening present and future societies [10,14].…”

“…As just alluded, a single heuristic, such as 'terminate all humans', or ethic, such as 'terminate all agents using resources inefficiently as defined by the following metric', added to a BCS could result in realization of the AGI existential threat, as could universal drives causing AGI to alter its utility function [14]. Thus, any alteration, especially forgery, of ethics modules or BCS must be detected.…”

“…The ability of AGI to self-correct or to assist its designers in correction of value alignment and behavior is called 'corrigibility' by Soares [21]. Miller et al review and examine how corrigibility can result in mis-alignment of values [14].…”

“…A systematic procedure could be followed as in Carlson ([10],Methods Sec. 2.1), testing BCS to flag and eliminate the possibility of behavioral pathways to dangerous AGI taken from enumerations given by, e.g., Asimov [60], Turchin [6], Bostrom [3], Yampolskiy [2], Tegmark [20]), and Miller et al [14].…”

Provably Safe Artificial General Intelligence via Interactive Proofs

“…As just alluded, a single heuristic, such as 'terminate all humans', or ethic, such as 'terminate all agents using resources inefficiently as defined by the following metric', added to a BCS could result in realization of the AGI existential threat, as could universal drives, such as simply wanting to improve its ability to achieve goals, causing AGI to alter its utility function [50]. Thus, any alteration, especially forgery, of ethics modules or BCS must be detected.…”