An Investigation of Challenges Encountered When Specifying Training Data and Runtime Monitors for Safety Critical ML Applications

Heyn, Hans-Martin; Knauss, Eric; Malleswaran, Iswarya; Dinakaran, Shruthi

doi:10.1007/978-3-031-29786-1_14

Cited by 6 publications

(1 citation statement)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(R2): Implement a systematic data selection process: This fosters "Trustworthiness by Design" by bridging the gap between the data selected for the training of the ML model and the requirement engineering process (L2) by establishing traceability mechanisms between system requirements, data requirements, and the completeness criteria [18].…”

Section: Rq3: Recommendations For Trustworthy (Er) Autonomous Systemsmentioning

confidence: 99%

RE-centric Recommendations for the Development of Trustworthy(er) Autonomous Systems

Ronanki,

Cabrero-Daniel,

Horkoff

et al. 2023

Proceedings of the First International Symposium on Trustworthy Autonomous Systems

View full text Add to dashboard Cite

Complying with the EU AI Act (AIA) guidelines while developing and implementing AI systems will soon be mandatory within the EU. However, practitioners lack actionable instructions to operationalise ethics during AI systems development. A literature review of different ethical guidelines revealed inconsistencies in the principles addressed and the terminology used to describe them. Furthermore, requirements engineering (RE), which is identified to foster trustworthiness in the AI development process from the early stages was observed to be absent in a lot of frameworks that support the development of ethical and trustworthy AI. This incongruous phrasing combined with a lack of concrete development practices makes trustworthy AI development harder. To address this concern, we formulated a comparison table for the terminology used and the coverage of the ethical AI principles in major ethical AI guidelines. We then examined the applicability of ethical AI development frameworks for performing effective RE during the development of trustworthy AI systems. A tertiary review and meta-analysis of literature discussing ethical AI frameworks revealed their limitations when developing trustworthy AI. Based on our findings, we propose recommendations to address such limitations during the development of trustworthy AI.

show abstract

Section: Rq3: Recommendations For Trustworthy (Er) Autonomous Systemsmentioning

confidence: 99%

RE-centric Recommendations for the Development of Trustworthy(er) Autonomous Systems

Ronanki,

Cabrero-Daniel,

Horkoff

et al. 2023

Proceedings of the First International Symposium on Trustworthy Autonomous Systems

View full text Add to dashboard Cite

show abstract

Requirements and software engineering for automotive perception systems: an interview study

Habibullah,

Heyn,

Gay

et al. 2024

Requirements Eng

View full text Add to dashboard Cite

Driving automation systems, including autonomous driving and advanced driver assistance, are an important safety-critical domain. Such systems often incorporate perception systems that use machine learning to analyze the vehicle environment. We explore new or differing topics and challenges experienced by practitioners in this domain, which relate to requirements engineering (RE), quality, and systems and software engineering. We have conducted a semi-structured interview study with 19 participants across five companies and performed thematic analysis of the transcriptions. Practitioners have difficulty specifying upfront requirements and often rely on scenarios and operational design domains (ODDs) as RE artifacts. RE challenges relate to ODD detection and ODD exit detection, realistic scenarios, edge case specification, breaking down requirements, traceability, creating specifications for data and annotations, and quantifying quality requirements. Practitioners consider performance, reliability, robustness, user comfort, and—most importantly—safety as important quality attributes. Quality is assessed using statistical analysis of key metrics, and quality assurance is complicated by the addition of ML, simulation realism, and evolving standards. Systems are developed using a mix of methods, but these methods may not be sufficient for the needs of ML. Data quality methods must be a part of development methods. ML also requires a data-intensive verification and validation process, introducing data, analysis, and simulation challenges. Our findings contribute to understanding RE, safety engineering, and development methodologies for perception systems. This understanding and the collected challenges can drive future research for driving automation and other ML systems.

show abstract

An empirical investigation of challenges of specifying training data and runtime monitors for critical software with machine learning and their relation to architectural decisions

Heyn,

Knauss,

Malleswaran

et al. 2024

Requirements Eng

View full text Add to dashboard Cite

The development and operation of critical software that contains machine learning (ML) models requires diligence and established processes. Especially the training data used during the development of ML models have major influences on the later behaviour of the system. Runtime monitors are used to provide guarantees for that behaviour. Runtime monitors for example check that the data at runtime is compatible with the data used to train the model. In a first step towards identifying challenges when specifying requirements for training data and runtime monitors, we conducted and thematically analysed ten interviews with practitioners who develop ML models for critical applications in the automotive industry. We identified 17 themes describing the challenges and classified them in six challenge groups. In a second step, we found interconnection between the challenge themes through an additional semantic analysis of the interviews. We explored how the identified challenge themes and their interconnections can be mapped to different architecture views. This step involved identifying relevant architecture views such as data, context, hardware, AI model, and functional safety views that can address the identified challenges. The article presents a list of the identified underlying challenges, identified relations between the challenges and a mapping to architecture views. The intention of this work is to highlight once more that requirement specifications and system architecture are interlinked, even for AI-specific specification challenges such as specifying requirements for training data and runtime monitoring.

show abstract

An Investigation of Challenges Encountered When Specifying Training Data and Runtime Monitors for Safety Critical ML Applications

Cited by 6 publications

References 44 publications

RE-centric Recommendations for the Development of Trustworthy(er) Autonomous Systems

RE-centric Recommendations for the Development of Trustworthy(er) Autonomous Systems

Requirements and software engineering for automotive perception systems: an interview study

An empirical investigation of challenges of specifying training data and runtime monitors for critical software with machine learning and their relation to architectural decisions

Contact Info

Product

Resources

About