Efficient, nearly orthogonal-and-balanced, mixed designs: an effective way to conduct trade-off analyses via simulation

Abstract

Designed experiments are powerful methodologies for gaining insights into the behaviour of complex simulation models. In recent years, many new designs have been created to address the large number of factors and complex response surfaces that often arise in simulation studies, but handling discrete-valued or qualitative factors remains problematic. We propose a framework for generating a design, of specified size, that is nearly orthogonal and nearly balanced for any mix of factor types (categorical, numerical discrete, and numerical continuous) and mix of factor levels. These new designs allow decision makers structured methods for trade-off analyses in situations that are not necessarily amenable to other methods for choosing alternatives, such as simulation optimization or ranking and selection approaches. These new designs also compare well to existing approaches for constructing custom designs for smaller experiments, and may also be of interest for exploring computer models in domains where fewer factors are involved.

Changing the paradigm: Simulation, now a method of first resort

“…Attempting to explore all possible combinations of 100 input factors, each at just two levels, for a simulation that runs as fast as a single elementary operation, would require all the Tianhe‐2's processing power for over 935 millenia. In contrast, even for a simulation with a run time measured in minutes and that has hundreds of factors and higher‐order effects, analysts can use efficient designs like those in Vieira et al to complete the necessary runs over a weekend using a single multicore desktop computer, or in a few hours or minutes on a high‐performance computing cluster .…”

Simulation experiments: Better data, not just big data

“…Similarly, we believe robust (resilient) alternatives found by seeking those that perform well over a variety of noise factor settings may be much more useful in practice than alternatives found by optimizing a single performance measure while implicitly holding noise factors constant. Finally, in trade-off analyses it may be much more useful to identify the existence of "knees in the curve," where we start to see increasing or decreasing rates of return, than to numerically predict exactly where these inflection points fall (Vieira Jr et al 2013). The interest in more complex questions may increase even more rapidly in the future (Elmegreen, Sanchez, and Szalay 2014).…”

“…If the experiment will investigate a mix of continuous-valued and discrete-valued factors, the nearly orthogonal-and-balanced (NOAB) designs of Vieira Jr et al (2013) are quite flexible. Code for constructing these designs is available, though it can take a long time to find a good design for large problems.…”

Testing aircraft fleet management policies using simulation experiments

“…We use a custom nearly orthogonal-and-balanced design (Vieira Jr. et al, 2013) with 1040 design points for the 19 factors, with 50 replications at each design point. Continuous factors are varied at 512 distinct levels.…”