Artificial Intelligence as a positive and negative factor in global risk

Abstract

By far the greatest danger of Artificial Intelligence (AI) is that people conclude too early that they understand it. Of course, this problem is not limited to the field of AI. Jacques Monod wrote: ‘A curious aspect of the theory of evolution is that everybody thinks he understands it’ (Monod, 1974). The problem seems to be unusually acute in Artificial Intelligence. The field of AI has a reputation for making huge promises and then failing to deliver on them. Most observers conclude that AI is hard, as indeed it is. But the embarrassment does not stem from the difficulty. It is difficult to build a star from hydrogen, but the field of stellar astronomy does not have a terrible reputation for promising to build stars and then failing. The critical inference is not that AI is hard, but that, for some reason, it is very easy for people to think they know far more about AI than they actually do. It may be tempting to ignore Artificial Intelligence because, of all the global risks discussed in this book, AI is probably hardest to discuss. We cannot consult actuarial statistics to assign small annual probabilities of catastrophe, as with asteroid strikes. We cannot use calculations from a precise, precisely confirmed model to rule out events or place infinitesimal upper bounds on their probability, as with proposed physics disasters. But this makes AI catastrophes more worrisome, not less. The effect of many cognitive biases has been found to increase with time pressure, cognitive busyness, or sparse information. Which is to say that the more difficult the analytic challenge, the more important it is to avoid or reduce bias. Therefore I strongly recommend reading my other chapter (Chapter 5) in this book before continuing with this chapter. When something is universal enough in our everyday lives, we take it for granted to the point of forgetting it exists. Imagine a complex biological adaptation with ten necessary parts. If each of the ten genes is independently at 50% frequency in the gene pool – each gene possessed by only half the organisms in that species – then, on average, only 1 in 1024 organisms will possess the full, functioning adaptation.

Putting the humanity into inhuman systems: How human factors and ergonomics can be used to manage the risks associated with artificial general intelligence

“…AGI is a dual use technology in that it will be used both for good and bad. First and foremost, if AGI realizes its potential and surpasses human intelligence, there is no doubt that it could bring significant benefits to humanity (Bostrom, 2014; Yudkowsky, 2008; 2012). Postulated benefits relate mainly to systems which exceed human intelligence and develop a capacity to respond to the panoply of issues that threaten either human health and wellbeing, the earth, or our future existence globally.…”

Meaningful Human Control over Autonomous Systems: A Philosophical Account

“…In this case, the system demonstrated its inability to track some morally relevant state of affairs: a rocket being “friendly” as opposed to just being of the kind usually used by allies; and because of that it was also unable to track the relevant moral reasons of the human commanders: targeting enemy rockets rather than just targeting rockets with certain material features. More recently, the use of machine-learning systems has allegedly led to misclassification of enemy and friendly tanks because the training set had many images of enemy tanks with clouds and many of friendly tanks with cloudless skies or tracking higher or lower resolution (Yudowsky, 2006 ). These systems were also tracking irrelevant properties in the training set.…”

◾ Utility Function Security in Artificially Intelligent Agents

“…This "parasite" rule achieved a very high rating because it appeared to be partly responsible for anything good that happened in the system. (Omohundro, 2008) While the two historical examples are mostly interesting as proofs of concept, future AI systems are predicted to be self-modifying and superintelligent (Bostrom, 2006a(Bostrom, , 2006bYampolskiy, 2011;Yampolskiy & Fox, 2012;Yampolskiy, 2013;Yudkowsky, 2008) making preservation of their reward functions (aka utility functions) an issue of critical importance. A number of specific and potentially dangerous scenarios have been discussed regarding wireheading by sufficiently capable machines, they include the following: Direct stimulation.…”