Risk-Based Decision-Making Fallacies: Why Present Functional Safety Standards are Not Enough

Abstract

Functional safety of a system is the part of its overall safety, understood as freedom from unacceptable/unreasonable risks that depends on a system operating correctly in response to its inputs. Functional safety elements are examined at every stage of the software development life cycle, including requirement specification, design, implementation, verification, validation and deployment. Acceptability of risks is judged within a framework of analysis with contextual and cultural aspects by individuals who may introduce subjectivity and misconceptions in the assessment. While functional safety standards elaborate much on the avoidance of unreasonable risk, little is addressed on the issue of avoiding unreasonable judgments of risk. Through the studies of common fallacies in risk perception and ethics, we present a moral-psychological analysis of functional safety standards and propose plausible improvements of the involved risk-related decision making processes, with a focus on the notion of an acceptable residual risk. As a functional safety reference model, we use the functional safety standard ISO 26262, addressing potential hazards caused by malfunctions of hardware and software systems within road vehicles, and defines safety measures that are required to achieve an acceptable level of safety. Analysis points out the critical importance of a robust safety culture with developed countermeasures to the common fallacies in risk perception, which are not addressed by contemporary functional safety standards. We argue that functional safety standards should be complemented with the analysis of potential hazards caused by fallacies in risk perception, their countermeasures, and the requirement that residual risks must be explicated, motivated, and accompanied by a plan for their continuous reduction. This approach becomes especially important in contemporary developed autonomous vehicles with increasing computational applications. I.

Ethical and Social Aspects of Self-Driving Cars

“…The approach based on "learning by experience" and "Proven in use" argument [1,4,53] presupposes a functioning socio-technological assurance system that has strong coupling among legislation, guidelines, standards and use, and promptly adapts to lessons learned. Ethical analysis in [22,38,60] addresses this problem of establishing and maintaining a functioning learning socio-technological system, while [38] discusses why functional safety standards are not enough.…”

Software Architecture Proposal for Navigation and Decisions of Autonomous Ground Vehicles in Controlled Environments with Unavoidable Collision Scenarios

“…However, the implementation of this rules is not a trivial problem, furthermore it carries a series of technological and ethical problems on its own, some authors like Andreas Johnsen et al [7], and Siddartha Khastgir et al [8], proposed more developed implementation of currently available functional safety standards, in both cases using as a base the standar ISO26262, in order to minimize the "risk" in the decision making process based on ethical rules of autonomous vehicles. In this paper, an approach to the problem of integrating an architecture that considers the navigation, external factors and a decision process based on ethical rules will be presented.…”