Collaborative Artificial Intelligence Needs Stronger Assurances Driven by Risks

Abstract

Collaborative AI systems (CAISs) aim at working together with humans in a shared space to achieve a common goal. This critical setting yields hazardous circumstances that could harm human beings. Thus, building such systems with strong assurances of compliance with requirements, domain-specific standards and regulations is of greatest importance. Only few scale impact has been reported so far for such systems since much work remains to manage possible risks. We identify emerging problems in this context and then we report our vision, as well as the progress of our multidisciplinary research team composed of software/systems, and mechatronics engineers to develop a risk-driven assurance process for CAISs.

Risk-driven Online Testing and Test Case Diversity Analysis for ML-enabled Critical Systems

“…Here, we introduce our case study used to illustrate the main steps of our approach, but also as a system subject in our empirical evaluation. The case study is an industrial CAIS used to carry out a production-relevant collaborative "pick and place" task along with a human operator [1], [15]. Figure 1a illustrates a high-level schema of the collaborative task, whereas Fig.…”

“…Autonomous systems running in safety-critical settings (e.g., collaborative robots, autonomous vehicles) increasingly rely on machine learning (ML) components (e.g., Deep Neural Networks) to mimic aspects of human intelligence, such as visionrelated tasks (e.g., image classification, and object detection). Collaborative robots, sometimes referred to as Collaborative Artificial Intelligence Systems (CAISs) [1], [2], belong to this class. The core functions of these systems are enabled by ML components to work together with humans in a shared physical space and achieve a common goal.…”

“…We conducted an empirical evaluation of our approach by using a CAIS case study simulated using the industrystrength robot simulator COPPELIASIM [9]. Our experiments aim at studying: (1) the effectiveness of the risk-driven test case generation across different search strategies and the statistical significance of the results; (2) the diversity of the generated test cases causing hazards. We found that selecting a particular testing strategy based on failure-revealing ability only may be misleading.…”