Assuring the Safety of Machine Learning for Pedestrian Detection at Crossings

Gauerhof, Lydia; Hawkins, Richard; Picardi, Chiara; Paterson, Colin; Hagiwara, Yuki; Habli, Ibrahim

doi:10.1007/978-3-030-54549-9_13

Cited by 33 publications

(21 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous work on the safety assurance of machine learning has focused on the structure of the assurance case and associated processes with respect to exist-ing safety standards [6,27,16,3,10]. Other work has focused on the effectiveness of specific metrics and measures on providing meaningful statements related to safety properties of the ML function [11,20,29,12].…”

Section: Related Workmentioning

confidence: 99%

A causal model of safety assurance for machine learning

Burton¹

2022

Preprint

View full text Add to dashboard Cite

This paper proposes a framework based on a causal model of safety upon which effective safety assurance cases for ML-based applications can be built. In doing so, we build upon established principles of safety engineering as well as previous work on structuring assurance arguments for ML. The paper defines four categories of safety case evidence and a structured analysis approach within which these evidences can be effectively combined. Where appropriate, abstract formalisations of these contributions are used to illustrate the causalities they evaluate, their contributions to the safety argument and desirable properties of the evidences. Based on the proposed framework, progress in this area is re-evaluated and a set of future research directions proposed in order for tangible progress in this field to be made.

show abstract

Section: Related Workmentioning

confidence: 99%

A causal model of safety assurance for machine learning

Burton¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Methods presented earlier are cited as being a way to tackle precise problems, but they also make the case for more traditional techniques such as; redundancy and fault tolerance subsystems, look ahead components, backup systems, coverage criteria or traceability through collection of Neural Networks related artifacts such as weights or versions. For instance, [71] proposed an approach where the requirements of a ML model drives the safety assurance process for the ML model. The process splits into 5 stages: requirements elicitation, data management, model learning, model verification and model deployment.…”

Section: Direct Certificationmentioning

confidence: 99%

“…-NN-dependability-kit [39] is a data driven toolbox that aims to provide directives to ensure uncertainty reduction in all the life cycle steps, notably robustness analysis with perturbation metrics and t-way coverage. This technique was used by [71] mentioned in the previous section with its requirements-driven safety assurance approach for ML. Building upon NN-dependability-kit, "specific requirements that are explicitly and traceably linked to system-level safety analysis" were tackled.…”

Section: Othersmentioning

confidence: 99%

How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review

Tambon,

Laberge,

et al. 2021

Preprint

View full text Add to dashboard Cite

Context: Machine Learning (ML) has been at the heart of many innovations over the past years. However, including it in so-called "safety-critical" systems such as automotive or aeronautic has proven to be very challenging, since the shift in paradigm that ML brings completely changes traditional certification approaches.Objective: This paper aims to elucidate challenges related to the certification of ML-based safety-critical systems, as well as the solutions that are proposed in the literature to tackle them, answering the question "How to Certify Machine Learning Based Safety-critical Systems?".Method: We conduct a Systematic Literature Review (SLR) of research papers published between 2015 to 2020, covering topics related to the certification of ML systems. In total, we identified 217 papers covering topics considered to be the main pillars of ML certification: Robustness, Uncertainty, Explainability, Verification, Safe Reinforcement Learning, and Direct Certification. We analyzed the main trends and problems of each sub-field and provided summaries of the papers extracted.Results: The SLR results highlighted the enthusiasm of the community for this subject, as well as the lack of diversity in term of datasets and type of ML models. It also emphasized the need to further develop connections between academia and industries to deepen the domain study. Finally, it also illustrated This work is supported by the DEEL project CRDPJ 537462-18 funded by the National Science and Engineering Research Council of Canada (NSERC) and the Consortium for Research and Innovation in Aerospace in Québec (CRIAQ), together with its industrial partners Thales Canada inc, Bell Textron Canada Limited, CAE inc and Bombardier inc.

show abstract

“…Deep neural network (DNN) image classifiers are increasingly being proposed for use in safety critical applications [6,15,19,24], where their accuracy is quoted as close to, or exceeding, that of human operators [3]. It has been shown, however, that when the inputs to the classifier are subjected to small perturbations, even highly accurate DNNs can produce erroneous results [8,9,30].…”

Section: Introductionmentioning

confidence: 99%

DeepCert: Verification of Contextually Relevant Robustness for Neural Network Image Classifiers

Paterson¹,

Wu²,

Grese³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We introduce DeepCert, a tool-supported method for verifying the robustness of deep neural network (DNN) image classifiers to contextually relevant perturbations such as blur, haze, and changes in image contrast. While the robustness of DNN classifiers has been the subject of intense research in recent years, the solutions delivered by this research focus on verifying DNN robustness to small perturbations in the images being classified, with perturbation magnitude measured using established Lp norms. This is useful for identifying potential adversarial attacks on DNN image classifiers, but cannot verify DNN robustness to contextually relevant image perturbations, which are typically not small when expressed with Lp norms. DeepCert addresses this underexplored verification problem by supporting: (1) the encoding of real-world image perturbations; (2) the systematic evaluation of contextually relevant DNN robustness, using both testing and formal verification; (3) the generation of contextually relevant counterexamples; and, through these, (4) the selection of DNN image classifiers suitable for the operational context (i) envisaged when a potentially safety-critical system is designed, or (ii) observed by a deployed system. We demonstrate the effectiveness of DeepCert by showing how it can be used to verify the robustness of DNN image classifiers build for two benchmark datasets to multiple contextually relevant perturbations.

show abstract

Assuring the Safety of Machine Learning for Pedestrian Detection at Crossings

Cited by 33 publications

References 11 publications

A causal model of safety assurance for machine learning

A causal model of safety assurance for machine learning

How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review

DeepCert: Verification of Contextually Relevant Robustness for Neural Network Image Classifiers

Contact Info

Product

Resources

About