Driver error or designer error: Using the Perceptual Cycle Model to explore the circumstances surrounding the fatal Tesla crash on 7th May 2016

Algorithmic Decision-Making and the Control Problem

“…For instance, in low-level partially automated systems, such as SAE Level 1 autonomous vehicles, there is "a clear partition in task allocation between the driver and vehicle subsystems" (Banks et al 2018b, p. 283). As the level of automation increases, however, this allocation gets blurred to the point that drivers find it difficult forming accurate assessments of the vehicle's capabilities, and on the whole are inclined to overestimate them (Banks et al 2018b). Counteracting this effect may be the greater readiness to believe that a smaller, less sophisticated device-with fewer working parts and opportunities for system glitches-will be compensatingly less temperamental.…”

“…What we shall term "the control problem" arises from the tendency of the human agent within a human-machine control loop to become complacent, overreliant or unduly diffident when faced with the outputs of a reliable autonomous system. Although it might be thought innocuous, decades of research confirm that the problem is actually pernicious, and perhaps even intractable (Banks et al 2018b;Cunningham and Regan 2018;Greenlee et al 2018). Somewhat alarmingly, it seems to afflict experts as much as novices, and is largely resistant to training (see Parasuraman and Manzey 2010 for reviews).…”

“…We argue that, except in certain special circumstances (in which great care must be taken), algorithmic decision tools should not be used in high-stakes or safety-critical decisions unless the systems concerned are significantly "better than human" in the relevant domain or subdomain of decision-making-a position towards which an increasing number of human factors experts have somewhat reluctantly been driven: see e.g. Banks et al (2018b), Cunningham and Regan (2018), Walker et al (2015), Cebon (2015). More concretely, we recommend three strategies to address the control problem, the most promising of which involves a complementary (and potentially dynamic) coupling between highly proficient algorithmic tools and human agents working alongside one another.…”

Thematic issue: driving automation and autonomy

“…Over the past twenty years all of these predicted shortfalls have come to fruition. This has been shown most recently in the Tesla collision with the semi-trailer (Banks et al, 2018b) and the Uber collision with the cyclist. In one of the earliest published papers on vehicle automation, automated driving was show to both dramatically reduced workload and the readiness for the driver to intervene in emergencies (Stanton, Young and McCaulder, 1997).…”

An explainable multi-attribute decision model based on argumentation

“…by self-driving cars), it is important that human users understand the rationale behind decisions (Dong et al, 2017a) so that they may trust the system enough to delegate their task to it. Actually, trust can be seen as the glue that holds human users and systems together and the lubricant for tasks to be carried out smoothly, with systems' failures, such as Google autonomous car's crash (Mathur, 2015) and Tesla autopilot fatal car's crash (Banks et al, 2017), generating mistrust. Human trust can be strengthened by an understanding of why an intelligent system acts in a certain way, or comes to reach a certain conclusion, and explanations are means to facilitate this understanding.…”