Designed Experiments for the Defense Community

Abstract

The areas of application for design of experiments principles have evolved, mimicking the growth of U.S. industries over the last century, from agriculture to manufacturing to chemical and process industries to the services and government sectors. In addition, statistically based quality programs adopted by businesses morphed from total quality management to Six Sigma and, most recently, statistical engineering (see Hoerl and Snee 2010). The good news about these transformations is that each evolution contains more technical substance, embedding the methodologies as core competencies, and is less of a ''program.'' Design of experiments is fundamental to statistical engineering and is receiving increased attention within large government agencies such as the National Aeronautics and Space Administration (NASA) and the Department of Defense. Because test policy is intended to shape test programs, numerous test agencies have experimented with policy wording since about 2001. The Director of Operational Test & Evaluation has recently (2010) published guidelines to mold test programs into a sequence of well-designed and statistically defensible experiments. Specifically, the guidelines require, for the first time, that test programs report statistical power as one proof of sound test design. This article presents the underlying tenents of design of experiments, as applied in the Department of Defense, focusing on factorial, fractional factorial, and response surface design and analyses. The concepts of statistical modeling and sequential experimentation are also emphasized. Military applications are presented for testing and evaluation of weapon system acquisition, including force-on-force tactics, weapons employment and maritime search, identification, and intercept.KEYWORDS factorial design, optimal design, power, response surface methodology, space filling design, test and evaluation WHY DOES THE DEFENSE COMMUNITY NEED DESIGN OF EXPERIMENTS?Any organization serious about testing should embrace methods and a general strategy that will cover the range of product employment, extract the most information in limited trials, and identify parameters affecting

Energy-Efficiency Assessment and Improvement—Experiments and Analysis Methods

“…Curiously, most case studies reported on scientific journals and textbooks on DoE do not illustrate the full sequential nature of an investigation. In many problems, the authors justify their approach to solve the problem, and use two out of three experimental phases, namely, screening/characterization [44] and screening/optimization [45]. A comprehensive example of the sequential learning process in the response surface methodology framework, including the optimization of multiple responses simultaneously, is presented by Lv et al [46].…”

When should I stop experimenting? Sample size considerations in I‐optimal designs

“…Johnson et al present a number of military testing applications from a design of experiments perspective. For example, in Johnson et al, they discuss a five‐factor military force‐level encounter example and a seven‐factor air‐to‐air missile simulation example. In both experiments, the cost of experimentation is high.…”

Conflict: Operational Realism versus Analytical Rigor in Defense Modeling and Simulation

“…The ability to relate the results to actual operations was provided through the use of a realistic mission scenario, and the use of current and qualified weapon system operators in the experiment. This experiment also adhered to an accepted factorial design [7] process.…”