Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurance

Abstract

The increasing use of Machine Learning (ML) components embedded in autonomous systems—so-called Learning-Enabled Systems (LESs)—has resulted in the pressing need to assure their functional safety. As for traditional functional safety, the emerging consensus within both, industry and academia, is to use assurance cases for this purpose. Typically assurance cases support claims of reliability in support of safety, and can be viewed as a structured way of organising arguments and evidence generated from safety analysis and reliability modelling activities. While such assurance activities are traditionally guided by consensus-based standards developed from vast engineering experience, LESs pose new challenges in safety-critical application due to the characteristics and design of ML models. In this article, we first present an overall assurance framework for LESs with an emphasis on quantitative aspects, e.g., breaking down system-level safety targets to component-level requirements and supporting claims stated in reliability metrics. We then introduce a novel model-agnostic Reliability Assessment Model (RAM) for ML classifiers that utilises the operational profile and robustness verification evidence. We discuss the model assumptions and the inherent challenges of assessing ML reliability uncovered by our RAM and propose solutions to practical use. Probabilistic safety argument templates at the lower ML component-level are also developed based on the RAM. Finally, to evaluate and demonstrate our methods, we not only conduct experiments on synthetic/benchmark datasets but also scope our methods with case studies on simulated Autonomous Underwater Vehicles and physical Unmanned Ground Vehicles.

Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous Systems

“…In the following, we discuss how the above verification problem may contribute to the computation of reliability. Similar to [28], [29], we partition the space of initial states into d sets, each of which is represented as a constraint C i , for i = 1..d. Based on these, we can estimate the reliability (defined as the probability of failure in satisfying ϕ with the policy π in the environment E) as…”

“…where m represents the sample size in the summation, G θ (C i ) returns the probability density of the partition i that is represented as the constraint C i , x Ci denotes the central point (i.e., a representative) of C i , and 1−P r(M E (π, x Ci ), ϕ) returns the failure rate of the DRL agent π working on inputs satisfying the constraint C i under the environment E. Note that G θ can be estimated in the same way as the data distribution in [28], [29].…”

Research on data reliability and storage security verification for cloud platform based on machine learning

“…The integration of these technologies provides users with powerful tools to gain insights into data and drive innovation. For example, companies can use machine learning services on cloud platforms to analyze consumer behavior and optimize products and services [5] . However, the use of cloud platforms also comes with some challenges, especially when it comes to data security and privacy.…”

STPA for Learning-Enabled Systems: A Survey and A New Practice

“…Recently, the use of Machine Learning (ML) to analyse complex data and integrate them into Cyber-Physical Systems (CPSs), known as Learning-Enabled Systems (LESs), has become widespread. When applying LESs in safetycritical domains, safety assurance is essential to their successful deployment and regulatory compliance, which remains a pressing challenge [13], [34], [60] despite recent efforts [7], [8], [29], [20]. Hazard identification is the first and critical step in safety assurance which enables the detection of causes and measures for mitigations of safety risks.…”