Automatic simulation-based testing of autonomous ships using Gaussian processes and temporal logic

Abstract

A methodology for automatic simulation-based testing of control systems for autonomous vessels is proposed. The work is motivated by the need for increased test coverage and formalism in the verification efforts. It aims to achieve this by formulating requirements in the formal logic Signal Temporal Logic (STL). This enables automatic evaluation of simulations against requirements using the STL robustness metric, resulting in a robustness score for requirements satisfaction. Furthermore, the proposed method uses a Gaussian Process (GP) model for estimating robustness scores including levels of uncertainty for untested cases. The GP model is updated by running simulations and observing the resulting robustness, and its estimates are used to automatically guide the test case selection toward cases with low robustness or high uncertainty. The main scientific contribution is the development of an automatic testing method which incrementally runs new simulations until the entire parameter space of the case is covered to the desired confidence level, or until a case which falsifies the requirement is identified. The methodology is demonstrated through a case study, where the test object is a Collision Avoidance (CA) system for a small high-speed vessel. STL requirements for safety distance, mission compliance, and COLREG compliance are developed. The proposed method shows promise, by both achieving verification in feasible time and identifying falsifying behaviors which would be difficult to detect manually or using brute-force methods. An additional contribution of this work is a formalization of COLREG using temporal logic, which appears to be an interesting direction for future work.

Evolution of Safety in Marine Systems: From System-Theoretic Process Analysis to Automated Test Scenario Generation

“…A method for automatic simulation-based testing of control systems for autonomous vessels was proposed in [8]. The method formulated the requirements using formal Signal Temporal Logic (STL) and used a Gaussian Process (GP) model to estimate the robustness score and the uncertainty level for the untested cases in the simulation.…”

“…In this paper, the results from [5], [6] and [8] are combined. A high-level risk analysis is performed using STPA.…”

A method to identify and rank objects and hazardous interactions affecting autonomous ships navigation

“…The preceding challenge can be tackled through systematic identification and analysis of the potential collision and interaction scenarios, proper understanding of the ANS interactions with the other ship systems (Abaei et al, 2021), as well as by developing effective testing and verification techniques (Pedersen et al, 2020;Torben et al, 2022) which would allow for providing sufficient confidence in the effectiveness of the ANS functionalities under varying conditions. Hence, for the safety assurance of the MASS operations, an adequate number of scenarios, including the critical ones, must be first identified and subsequently tested in both virtual and real environments.…”

“…Hence, for the safety assurance of the MASS operations, an adequate number of scenarios, including the critical ones, must be first identified and subsequently tested in both virtual and real environments. As the number of the potential scenarios is expected to be enormous, a formal method is required to limit the number of these scenarios to the extent possible and to conclude on the ones that eventually must be tested (either in virtual or in real environments) (Pedersen et al, 2020;Torben et al, 2022).…”

“…As the COLREGS do not include numerical criteria for crew actions (Woerner et al, 2019;Abebe et al, 2021;Ni et al, 2021), their implementation relies on effective crew judgement (Du et al, 2020;Huang et al, 2020). Hence, the COLREGS cannot be employed to develop a comprehensive set of testing scenarios and requirements in encountering cases between the own MASS and surrounding objects (target ships and obstacles) (Woerner et al, 2019;Torben et al, 2022).…”