Cognitive biases potentially affecting judgement of global risks

Abstract

All else being equal, not many people would prefer to destroy the world. Even faceless corporations, meddling governments, reckless scientists, and other agents of doom, require a world in which to achieve their goals of profit, order, tenure, or other villainies. If our extinction proceeds slowly enough to allow a moment of horrified realization, the doers of the deed will likely be quite taken aback on realizing that they have actually destroyed the world. Therefore I suggest that if the Earth is destroyed, it will probably be by mistake. The systematic experimental study of reproducible errors of human reasoning, and what these errors reveal about underlying mental processes, is known as the heuristics and biases programme in cognitive psychology. This programme has made discoveries highly relevant to assessors of global catastrophic risks. Suppose you are worried about the risk of Substance P, an explosive of planet-wrecking potency which will detonate if exposed to a strong radio signal. Luckily there is a famous expert who discovered Substance P, spent the last thirty years working with it, and knows it better than anyone else in the world. You call up the expert and ask how strong the radio signal has to be. The expert replies that the critical threshold is probably around 4000 terawatts. ‘Probably?’ you query. ‘Can you give me a 98% confidence interval?’ ‘Sure’, replies the expert. ‘I’m 99%confident that the critical threshold is above 500 terawatts, and 99%confident that the threshold is below 80,000 terawatts.’ ‘What about 10 terawatts?’ you ask. ‘Impossible’, replies the expert. The above methodology for expert elicitation looks perfectly reasonable, the sort of thing any competent practitioner might do when faced with such a problem. Indeed, this methodology was used in the Reactor Safety Study (Rasmussen, 1975), now widely regarded as the first major attempt at probabilistic risk assessment. But the student of heuristics and biases will recognize at least two major mistakes in the method – not logical flaws, but conditions extremely susceptible to human error. I shall return to this example in the discussion of anchoring and adjustments biases (Section 5.7).

How argumentative writing stifles open-mindedness

“…That is, the more of an expert one is on the topic, the more their belief strengthened after being presented with evidence that both supported and contradicted their view. This cognitive bias is called the sophistication effect (Yudkowsky, 2008). The more knowledgeable one is on a subject, the more counterarguments they can utilize to rebut or explain away disconfirming evidence.…”

“… 9 Confirmation bias can also be either “hot” or “cold” or what Nickerson (1998) calls “motivated” or “unmotivated.” Hot confirmation bias concerns emotionally charged topics such as divisive moral and political questions to which individuals have a vested interest (Yudkowsky, 2008). By contrast, cold confirmation bias concerns emotionally neutral topics and the effect of the bias is less in these cases (Yudkowsky, 2008). Since argumentative research papers can often engage with emotionally charged topics, I am focusing on hot confirmation bias.…”

Tourist Aviation Emissions: A Problem of Collective Action

“…People routinely break the rules of probability claiming, for example, that the future risk of A+B is greater than the risk of A (Tversky and Kahneman 1974; for an application in the context of environmental policy, see Medvecky 2012). Moreover, assessments of likelihood are often influenced by the specificity or vividness with which phenomena are described (Yudkowsky 2008). As the dangers of global warming are diffuse and wide-ranging, people are likely to mentally discount them while accentuating the risks of vivid and specific (but unlikely) scenarios involving terrorism for example.…”

Artificial Intelligence as a positive and negative factor in global risk

“…It is a risky intellectual endeavor to predict specifically how a benevolent AI would help humanity, or an unfriendly AI harm it. There is the risk of conjunction fallacy: added detail necessarily reduces the joint probability of the entire story, but subjects often assign higher probabilities to stories which include strictly added details (Yudkowsky 2008).…”

“…In my other chapter for Global Catastrophic Risks, "Cognitive Biases Potentially Affecting Judgment of Global Risks" (Yudkowsky 2008), I opened by remarking that few people would deliberately choose to destroy the world; a scenario in which the Earth is destroyed by mistake is therefore very worrisome. Few people would push a button that they clearly knew would cause a global catastrophe.…”