896 19 037 IDA Memorandum M-496 is a related document which provides a comprehensive, annotated bibliography of the reference material acquired in the course of this work. IDA Memorandum M-513, another related document, is a collection of the papers provided by leading experts in testing and evaluation tech-* nology preparatory to an IDA workshop held in support of this work.The authors gratefully acknoledge the support given by all those who participated in the workshop and reviewed this paper. The insights and experience these people so willingly shared has been invaluable. In particular, special thanks go to the following people:
IntroductionTesting and uvaluation are recognized as pivotal problems in the development of the Strategic Defense System (SDS). Consequently, a development approach combining design, prototyping, and simulation is being used to allow early evaluation of system requirements and designs. Loosely-coupled, decentralized system architectures are being examined for their facility to reduce the testing problem to a manageable level. But what testing and evaluation technology shall be used to ensure the reliability of, and provide the necessary level of confidence in, SDS software? The purpose of this report is to address this question.
*For the past decade, software testing and evaluation have been relatively unpopular subjects. Why this happened is less important than its consequences. It has been a contributing factor to the lag between the state-of-the-art and state-of-the-practice, which is the largest of any in the diverse areas of software engineering, and has resulted in a shortage of research and development (R&D) resources and a small, weak research community. Since SDS prototype software is already being developed, software testing and evaluation are now critical concerns. The Strategic Defense Initiative Organization (SDIO) cannot * wait for testing and evaluation issues to pop up later in the development process (a traditional approach which has proven costly in the past), but must meet the challenge up front.Once the importance of this challenge is recognized, appropriate actions can be taken. There is a substantial body of testing and evaluation technology whose use in an industrial environment is ready to be investigated, preparatory to transitioning the technology into practice. The main body of this report * describes the current status of different areas of this technology. It also outlines a number of tasks for bringing promising elements of the technology into practice. While the use of advanced technology will certainly help, it is by no means sufficient to resolve all testing and evaluation problems. In particular, technology for the testing and evaluation of large, distributed and real-time software systems is still in its infancy. The final part of this report, therefore, recommends a number of tasks to extend the boundaries of technology to meet SDS needs.Testing and evaluation technology is not, by itself, enough to ensure improved practices. Policy must evolve ...