SDS (Strategic Defense System) Software Testing and Evaluation: a Review of the State-of-the-Art in Software Testing and Evaluation with Recommended R&D Tasks

Abstract

896 19 037 IDA Memorandum M-496 is a related document which provides a comprehensive, annotated bibliography of the reference material acquired in the course of this work. IDA Memorandum M-513, another related document, is a collection of the papers provided by leading experts in testing and evaluation tech-* nology preparatory to an IDA workshop held in support of this work.The authors gratefully acknoledge the support given by all those who participated in the workshop and reviewed this paper. The insights and experience these people so willingly shared has been invaluable. In particular, special thanks go to the following people: IntroductionTesting and uvaluation are recognized as pivotal problems in the development of the Strategic Defense System (SDS). Consequently, a development approach combining design, prototyping, and simulation is being used to allow early evaluation of system requirements and designs. Loosely-coupled, decentralized system architectures are being examined for their facility to reduce the testing problem to a manageable level. But what testing and evaluation technology shall be used to ensure the reliability of, and provide the necessary level of confidence in, SDS software? The purpose of this report is to address this question. *For the past decade, software testing and evaluation have been relatively unpopular subjects. Why this happened is less important than its consequences. It has been a contributing factor to the lag between the state-of-the-art and state-of-the-practice, which is the largest of any in the diverse areas of software engineering, and has resulted in a shortage of research and development (R&D) resources and a small, weak research community. Since SDS prototype software is already being developed, software testing and evaluation are now critical concerns. The Strategic Defense Initiative Organization (SDIO) cannot * wait for testing and evaluation issues to pop up later in the development process (a traditional approach which has proven costly in the past), but must meet the challenge up front.Once the importance of this challenge is recognized, appropriate actions can be taken. There is a substantial body of testing and evaluation technology whose use in an industrial environment is ready to be investigated, preparatory to transitioning the technology into practice. The main body of this report * describes the current status of different areas of this technology. It also outlines a number of tasks for bringing promising elements of the technology into practice. While the use of advanced technology will certainly help, it is by no means sufficient to resolve all testing and evaluation problems. In particular, technology for the testing and evaluation of large, distributed and real-time software systems is still in its infancy. The final part of this report, therefore, recommends a number of tasks to extend the boundaries of technology to meet SDS needs.Testing and evaluation technology is not, by itself, enough to ensure improved practices. Policy must evolve ...

Software Testing Initiative for Strategic Defense Systems

“…However, previous studies by the Institute for Defense Analyses (IDA) have concluded that conventional testing methods sufficient for small-scale sequential software may not be adequate for testing software for strategic defense systems [Youngblut 1989, Brykczynski 1990. Moreover, there is no concerted effort to ensure that advanced methods for testing software are used within the SDI program.…”

Software Inspection Insertion Efforts for the Ballistic Missile Defense Organization

“…In the late 1980's, the Strategic Defense Initiative Organization (SDIO) tasked IDA to examine the technology needed for testing the software for a Strategic Defense System 0 (SDS). An assessment of the state of the art in software testing techniques and their potential applicability to SDS software was performed [Youngblut 1989]. In 1991, IDA produced a paper that described an SDIO initiative designed to develop and deploy needed software testing technology to ensure the development of reliable and cost-effective software for the S Strategic Defense Initiative (SDI) program [Brykczynski 19921.…”