Perceptions on the State of the Art in Verification and Validation in Cyber-Physical Systems

Abstract

It is widely held that debugging cyber-physical systems (CPS) is challenging; many strongly held beliefs exist regarding how CPS are currently debugged and tested and the suitability of various techniques. For instance, dissenting opinions exist as to whether formal methods (including static analysis, theorem proving, and model checking) are appropriate in CPS verification and validation. Simulation tools and simulation-based testing are also often considered insufficient for CPS. Many "experts" posit that high-level programming languages (e.g., Java or C#) are not applicable to CPS due to their inability to address (significant) resource constraints at a high level of abstraction. To date, empirical studies investigating these questions have not been done. In this paper, we qualitatively and quantitatively analyze why debugging CPS remains challenging and either dispel or confirm these strongly held beliefs along the way. Specifically, we report on a structured online survey of 25 CPS researchers (10 participants classified themselves as CPS developers), semistructured interviews with nine practitioners across four continents, and a qualitative literature review. We report these results and discuss several implications for research and practice related to CPS.

A Comprehensive Technological Survey on Dependable Self-Managing CPS: The Decade of Researches on Correctness and Dependability

“…Lots of testing and formal methods have been proposed which involve from consistency verification [76] to dynamic logic validation [77], and the targets include circuits, chips, network, software, systems and requirements, etc. [76][77][78][79]. Sara et al analyzed the need of multiple levels testing on CPS and made the state-of-the-art survey on testing methods and conclude the challenges in CPS testing [80].…”

“…E.g. worstcase execution time (WCET) analysis [104], perceptions scheduling V&V [79]. A general model of mixed-criticality recurrent real-time tasks was proposed, and the analysis model took WCET, relative deadline, and period into account at two criticality levels [105].…”

BraceAssertion: Runtime Verification of Cyber-Physical Systems

“…Compared with the growth of the domain, verification and validation of CPS lags far behind [32]. The state of the practice in debugging CPS generally entails a combination of simulation and in situ debugging.…”

“…The analysis concluded that, although simulation-centric tools are indispensable for rapid prototyping, design, and debugging, they are limited in providing correctness guarantees. State of the art formal methods tools, including static analysis, theorem proving, and model checking, are insufficient in tackling the challenges in CPS verification and validation [32]. Other verification techniques, including model-based testing [11] and simulation [16] have high learning curves, impractical development costs, and scalability issues.…”

“…INTRODUCTION Cyber-Physical Systems (CPS) are found in applications in structural monitoring, autonomous vehicles, and many other fields. Compared with the growth of the domain, verification and validation of CPS lags far behind [32]. The state of the practice in debugging CPS generally entails a combination of simulation and in situ debugging.…”