Practical Considerations in Selecting the Best Set of Simulated Systems

Abstract

In many practical problems, simulation models are used to support complex decision-making processes comparing hundreds or thousands of solutions. These problems typically have a key objective but the final decision may be dependent on other factors, which cannot be incorporated into the simulation model. In such cases, decision-makers may request a short list of 'good' solutions, which work well for the main objective and satisfy one or more chance constraints. While fully sequential ranking and selection procedures can be effective at solving these problems, surveys of experimentation practice suggest that they are under-utilized, potentially due to difficulties automating commercial software. We develop an approach with just two stages of replications. The approach, which has been designed to cope with the use of common random numbers, draws on ideas from indifference zones and makes use of bootstrapping to find a subset of high quality solutions. A Python implementation is freely available.

A Practical Approach to Subset Selection for Multi-objective Optimization via Simulation

“…This article extends the work described in [Monks and Currie 2018] in which we introduced the basic ideas of the method, with the most significant advance being a thorough comparison of its performance with OCBA-m on a standard set of problems frequently used to measure performance. We also extend the treatment of the hospital simulation model that was considered previously.…”