Comprehending software correctness implies comprehending an intelligence-related limitation

Abstract

This article applies mathematical logic to obtain a rigorous foundation for previous inherently nonrigorous results and also extends those previous results. Roughly speaking, our main theorem states: any agent A that comprehends the correctness-related properties of software S also comprehends an intelligence-related limitation of S. The theorem treats the output of S, if any, as an attempt at solving a halting problem. Previous nonrigorous attempts to obtain similar theorems depend on infallibility assumptions on both the agent and the software. The hypothesis that intelligent agents and intelligent software must be infallible has been widely questioned. In addition, recent work by others has determined that well-known previous attempts use a fallacious form of reasoning; that is, the same form of reasoning can yield paradoxical results. Our main theorem avoids infallibility assumptions on both the agent and the software. In addition, our proof is rigorous, in the sense that in principle one can carry it out in Zermelo-Fraenkel set theory. The software correctness framework considered in the main theorem is that of Hoare logic.

The Comprehensibility Theorem and the Foundations of Artificial Intelligence

“…Perfect judgments about correctness may lie beyond human capabilities; the mathematical model described in ''The Mathematical Model'' is independent of that issue. This game is mentioned briefly-as the ''halting problem game''-in Charlesworth (2006), but that article lacks the motivation for the proof of the Comprehensibility Theorem provided by the non-rigorous argument in our next section.…”

“…Such codes also permit us to define an agent as simply a function from natural numbers to natural numbers, which does not restrict applications of the model, since any discretized input or output of a human, robot, or other system is a finite string of bits and one can view each finite string of bits as representing a natural number. This article-unlike (Charlesworth 2006)-eliminates nearly all explicit notation for the codes of Turing machines and formulas, via the use of simplified expository conventions explained in the next section. [It is possible to give a solid proof of a general version of Gödel's Theorem that entirely avoids numerical coding.…”

“…To avoid well-known pitfalls of paradoxical reasoning, crucially needed are mathematical definitions of such terminology as ''agent'', ''correctly deduce'', and ''adequate''. The next two sections provide what we hope is substantially more-helpful motivation for the definitions (and proof of the theorem), than that given by Charlesworth (2006). Those definitions then appear in ''The Mathematical Model'' section.…”

“…The Comprehensibility Theorem, published in a journal for specialists in formal mathematical logic (Charlesworth 2006), appears to be such a theorem. The present article examines the extent to which this impression withstands additional study, addressing many questions about the theorem.…”

“…Although the proof of the theorem in Charlesworth (2006) is entirely about the ability to make deductions, that article uses the word ''understand'' in the statement of the theorem. To achieve greater clarity the present article avoids using that word in the theorem.…”