Introducing CombinatorialTesting in a Large Organization I n 2009, when Lockheed Martin decided to explore the benefits of combinatorial testing methods as a way to reduce testing costs and maintain competitive processes, its decision was based on several developments. That same year, pilot projects using these methods at Eglin Air Force Base in Florida were showing promise in reducing testing costs. The US Department of Defense's (DoD's) Office of Test and Evaluation had recently endorsed the design of experiments (DOE) statistical testing approach, 1 some principles of which are in combinatorial testing. In the commercial realm, firms were reporting success with pairwise testing, a basic form of combinatorial testing that covers two-way factor combinations.Together, these developments were sufficient motivation for Lockheed Martin to launch its own pilot projects. The goal was to make combinatorial testing available to its engineers and use it internally without mandates from either customers or management. To aid in this effort, the company developed a cooperative research and development agreement (CRADA) with the National Institute of Standards and Technology (NIST), which is one of NIST's mechanisms for conducting joint research with US industry. CRADAs also provide flexibility in structuring projects, assigning intellectual property rights, and protecting industry-related proprietary information and research results.Lockheed Martin and NIST entered into the CRADA to better understand the applicability and effectiveness of a relatively new approach for software testing to improve the quality, safety, and reliability of US products and systems. A goal of particular interest was to better understand the challenges in introducing a new software-testing approach in a large US corporation.Specific goals in introducing the method in Lockheed Martin projects were to › evaluate the concept's viability; › test process improvement in a variety of domains, including system, software, and hardware testing; › make tests more effective in finding problems; and A two-year study of eight pilot projects to introduce combinatorial testing in a large aerospace corporation found that the new methods were practical, significantly lowered development costs, and improved test coverage by 20 to 50 percent.

No abstract

No abstract

The process used to validate, verify, and test flight avionics control systems has produced software that is highly reliable. However, ever greater demands for reliability require new automated tools to improve existing processes. We used the Anna (Annotated Ada) formal specification language and supporting tool set to develop a Test Range Oracle Tool (TROT) to automate the testing of equation execution. Our approach fits within the existing testing process, automates perviously manual analysis, and can increase the level of test coverage. The TROT approach also introduces the use of formal specification languages and supporting tools to an existing industry program. This approach supported production tests and is being expanded into other test support areas.

No abstract