Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction

Abstract

As debates about the policy and ethical implications of AI systems grow, it will be increasingly important to accurately locate who is responsible when agency is distributed in a system and control over an action is mediated through time and space. Analyzing several high-profile accidents involving complex and automated socio-technical systems and the media coverage that surrounded them, I introduce the concept of a moral crumple zone to describe how responsibility for an action may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system. Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions. While the crumple zone in a car is meant to protect the human driver, the moral crumple zone protects the integrity of the technological system, at the expense of the nearest human operator. The concept is both a challenge to and an opportunity for the design and regulation of human-robot systems. At stake in articulating moral crumple zones is not only the misattribution of responsibility but also the ways in which new forms of consumer and worker harm may develop in new complex, automated, or purported autonomous technologies.

Calibrating Agency: Human‐Autonomy Teaming and the Future of Work amid Highly Automated Systems

“…Yet teaming's feel-good ethos of 'bringing out the best in everyone,' and its promise of flexibility in designing interactional relationships, leaves room to stretch out into implications for ethical and political domainsespecially since design prescriptions such as "transparency," as we will see, operate deeply on both the functional and ethical levels. This has left us room to more quietly address issues brought up in the work of anthropologists of technological labor (Gray and Suri 2019;Elish 2019), especially the ethical consequences of calibrating agency.…”

“…In a recent paper in Data & Society, Elish (2019) describes that intelligent and autonomous systems in every form have the potential to generate "moral crumple zones." A "moral crumple zone" describes how responsibility for an automation error may be incorrectly displaced onto a human actor within the system who in fact had very little control over the erroneous behavior:…”

“…Yet the report and subsequent media coverage focused on the safety driver's behavior, with concerns raised as to whether the she was looking at her cell phone or streaming media (Somerville & Shepardson 2018). In other words, despite a complex set of factors precipitating the crash, public scrutiny focused on the driver, who may now be facing criminal charges (Elish 2019).…”

Robots in American Law

“…We may not wish to incentivize the preservation of human control, even where less safe or efficient, merely to furnish a human "crumple zone" for liability. 261 I am aware that the liability still winds up landing on one or more humansperhaps the manufacturer of the driverless car instead of whoever happens to be behind the skeuomorphic wheel. Even so, the metaphors and analogies we use influence which human pays the price for a robotic harm.…”

Human Judgement in Algorithmic Loops; Individual Justice and Automated Decision-Making

“…Another set of reasons for maintaining humans as the locus of decision-making power is that they are situated in social and institutional contexts which allow for liability and apportioning of responsibility (Bryson et al, 2017), or societal legitimation (Rahwan, 2018). According to this view, human involvement needs to be substantive to ensure that genuine thought has been applied, and that control can be exercised where necessary, otherwise these liability arrangements may risk humans being reduced to rubber-stamping quasi-automated decisions which they were not meaningfully involved in (Wagner, 2019), or acting as a 'moral crumple zone' (Elish, 2016). Even if algorithms could effectively learn from observing human judgements, we may still want to keep human decision makers around to generate fresh ground truth and to avoid moral atrophy (Hildebrandt, 2013).…”