Simulation-based Design Optimization for Statistical Power: Utilizing Machine Learning

Abstract

The planning of adequately powered research designs increasingly goes beyond determining a suitable sample size. More challenging scenarios demand simultaneous tuning of multiple design parameter dimensions and can only be tackled using Monte Carlo simulation in case that no analytical approach is available. Since there are usually several possible solutions, we are often interested in a specific solution that is optimal, for example, in terms of the financial cost of an experiment. We introduce a new surrogate modeling framework based around machine learning predictions to address these types of problems. Possible applications include finding design parameters that imply a desired power at minimum cost or, alternatively, maximum power at a cost threshold. As surrogate models, which form the basis for the prediction during the framework iterations, we implement Gaussian process regression and support vector regression. Our implementation is publicly available in an R package. We investigate the approach in an extensive simulation study using six different hypothesis test scenarios with single and multidimensional design parameters from the fields of multilevel modeling, item response theory, and others. The results demonstrate the high efficiency of this novel framework for simulation-based design optimization.

Sample Size Planning for Complex Study Designs: A Tutorial for the mlpwr Package

“…Also, in a grid search, the computational cost can be very high if we want to estimate the performance for a larger number of design parameter sets. Surrogate modeling can be more efficient than grid search even when there is only one design parameter: In a simple example, our approach required only 20% of the computational effort and performed 50% more simulation runs that used the optimal sample size (Zimmer & Debelak, 2022).…”

“…In a recent manuscript, we extended the surrogate modeling framework for multiple design parameter dimensions to directly include cost considerations during the search (Zimmer & Debelak, 2022). In this way, if there are multiple constellations of design parameter sets that all imply similar power, we can efficiently find the cost-optimal design among them.…”

“…We recently contributed the mlpwr R package to CRAN. It relies on the framework evaluated in Zimmer and Debelak (2022) with a few minor changes to the surrogate modeling algorithm (see the Appendix). It can be applied to a wide range of study designs and statistical hypotheses.…”