Introducing Combinatorial Testing in a Large Organization

Abstract

Introducing CombinatorialTesting in a Large Organization I n 2009, when Lockheed Martin decided to explore the benefits of combinatorial testing methods as a way to reduce testing costs and maintain competitive processes, its decision was based on several developments. That same year, pilot projects using these methods at Eglin Air Force Base in Florida were showing promise in reducing testing costs. The US Department of Defense's (DoD's) Office of Test and Evaluation had recently endorsed the design of experiments (DOE) statistical testing approach, 1 some principles of which are in combinatorial testing. In the commercial realm, firms were reporting success with pairwise testing, a basic form of combinatorial testing that covers two-way factor combinations.Together, these developments were sufficient motivation for Lockheed Martin to launch its own pilot projects. The goal was to make combinatorial testing available to its engineers and use it internally without mandates from either customers or management. To aid in this effort, the company developed a cooperative research and development agreement (CRADA) with the National Institute of Standards and Technology (NIST), which is one of NIST's mechanisms for conducting joint research with US industry. CRADAs also provide flexibility in structuring projects, assigning intellectual property rights, and protecting industry-related proprietary information and research results.Lockheed Martin and NIST entered into the CRADA to better understand the applicability and effectiveness of a relatively new approach for software testing to improve the quality, safety, and reliability of US products and systems. A goal of particular interest was to better understand the challenges in introducing a new software-testing approach in a large US corporation.Specific goals in introducing the method in Lockheed Martin projects were to › evaluate the concept's viability; › test process improvement in a variety of domains, including system, software, and hardware testing; › make tests more effective in finding problems; and A two-year study of eight pilot projects to introduce combinatorial testing in a large aerospace corporation found that the new methods were practical, significantly lowered development costs, and improved test coverage by 20 to 50 percent.

A Combinatorial Approach to Analyzing Cross-Site Scripting (XSS) Vulnerabilities in Web Application Security Testing

“…Combinatorial testing has been successfully applied for testing (critical) software systems in large organizations [11]. It is an already proven method for blackbox security testing of large-scale web software systems [7], [3], [2] where t-way testing was applied successfully to XSS detection.…”

Can Pairwise Testing Perform Comparably to Manually Handcrafted Testing Carried Out by Industrial Engineers?

“…One of the most commonly used strategy is pairwise (also known as two-way) testing in which each combination for all possible pairs of input parameters are covered by at least one test case. Several empirical studies [5], [6], [14], [15] on the use of combinatorial testing for industrial software have been reported and showed that pairwise testing is a very effective technique. In this paper we seek to investigate the use of pairwise testing for industrial control software and compare this technique with manual testing performed by industrial engineers.…”

“…Pairwise testing is an approach to combinatorial testing that generates a test suite which covers each combination of value pairs at least once. There is some evidence [5], [6] suggesting that pairwise testing is efficient and effective at detecting software faults. However, even if pairwise techniques have been found useful and applicable in industrial applications, the experimental evidence regarding its effectiveness in practice is still limited.…”

Locating Minimal Fault Interaction in Combinatorial Testing

“…Combinatorial testing could significantly reduce test cost and increase quality of software system [1]. It has been proved to be effective especially in a software system where faults come from the interactions of its parameters [2].…”

“…However, this method could only be used under the condition that the value's number of each parameter in software is not larger than 2. On the basis of LDA and ELA, Hagar et al [1] propose the method of Partial Covering Array (PCA) which could be used in the software with known safe value, and it presents a new combinatorial structure to generate ELA.…”