An evaluation of the optimality of frequent verification for vertically integrated systems

Abstract

Verification activities, such as inspection, testing, analysis, and demonstration, improve one's confidence in the system meeting the system requirements during the development process. Frequent verification is often advocated as a strategy that minimizes costs of rework over the entire design process, where frequent verification involves verifying after any change in the design. However, this strategy is yet to be validated. In this paper, we develop a belief‐based model of verification in systems design to determine the conditions under which frequent verification is an optimal strategy for a vertically integrated organization. Our model uses belief distributions to capture the organization's dynamic confidence in the system design meeting a requirement of interest during the development process. It also captures the organization's dynamic confidence in the correctness of its development activities (or design process) as a function of past verification activities and current system maturity. The analysis of our model shows that frequent verification is a cost‐minimizing strategy for any level of belief in satisfying the requirement only when the organization has high confidence in the correctness of its design activities and the expected cost to rework a faulty design is greater than the costs to set up the verification activities throughout the development process. Otherwise, strategies with infrequent verification are superior. Our work contributes to the growing body of literature on the theoretical foundations of systems engineering and engineering design and seeks to provide practitioners with a means to determine optimal verification strategies.

Modeling correction activities in the context of verification strategies

“…Use of synthetic data is considered adequate because actual predictions are not relevant to showcase the application of the modeling framework. This is an accepted practice in this type of research 13,14,77 . This BN captures the prior beliefs of all elements.…”

“…Despite the strong coupling between VAs and CAs, and the criticality of CAs, existing research focuses on the modeling and selection of just one type of activity, significantly simplifying or even ignoring the other activity. For example, CAs are treated as direct properties of VAs (not as independent activities) in research that aims to develop system verification models or to design methods to generate verification strategies 1,[12][13][14][15][16] . Similarly, VAs are ignored in work that aims to better understand CA models or analysis methods 10,17 .…”

Optimal verification strategies in multi‐firm projects

“…Yet, majority of the literature on verification activities adopts the aleatory uncertainty approach 3,5,21–37 . The reader is referred to ref 38,39 . for a discussion on these works and the drawbacks of adopting the aleatory uncertainty approach to model systems engineering projects.…”

“…Recent works on verification in systems engineering have begun to explore the benefits of modeling the epistemic uncertainty in systems engineering projects with belief distributions 38–47 . In these works, belief distributions are used to model the subjective confidence of designers in the true state of the system design.…”

“…In addition to exploring fundamental research questions on verification activities, a few of these belief‐based approaches to verification have exploited algorithmic approaches to determine optimal verification strategies in systems engineering projects 38,39,47–49 . These include the use of reinforcement learning 47 and dynamic programming 38,39 to explore the space of verification strategies in order to determine the optimal verifications strategy for a single firm. Our work contributes to this area of research by developing two initial concept models of verification in multi‐firm projects using belief distributions and developing exact algorithms to determine optimal verification for the same.…”