Sample Size Planning for Complex Study Designs: A Tutorial for the mlpwr Package

Abstract

A common challenge in designing empirical studies is determining an appropriate sample size. When more complex models are used, estimates of power can only be obtained using Monte Carlo simulations. In this tutorial, we introduce the R package mlpwr to perform simulation-based power analysis based on surrogate modeling. Surrogate modeling is a powerful tool to guide the search for study design parameters that imply a desired power or meet a cost threshold (e.g., in terms of monetary cost). mlpwr can be used to search for the optimal allocation when there are multiple design parameters, e.g., when balancing the number of participants and the number of groups in multilevel modeling. At the same time, the approach can take into account the cost of each design parameter, and aims to find a cost-efficient design. We introduce the basic functionality of the package, which can be applied to a wide range of statistical models and study designs. Additionally, we provide two examples based on empirical studies for illustration: one for sample size planning when using an item response theory model, and one for assigning the number of participants and the number of countries for a study using multilevel modeling.

Simulation-based design optimization for statistical power: Utilizing machine learning.

“…We implemented a tuning protocol for SVR that allows us to find suitable hyperparameters for a variety of scenarios. Our approach is implemented in the mlpwr R package (Zimmer & Debelak, 2022a;Zimmer, Henninger, et al, 2022).…”

“…As a unique contribution, our approach leads to increased efficiency by considering the cost of design parameters during the search. We provide an R package on the Comprehensive R Archive Network (Zimmer & Debelak, 2022a) that implements this new toolbox to help researchers solve complex design optimization problems in an efficient and reproducible way. Based on good results in pilot simulations, we implemented GPR and support vector regression (SVR) as surrogate models, where the latter, to our knowledge, has not yet been applied to surrogate modeling for sample size planning.…”

“…All analysis codes and the online supplemental materials for this study are available at the Open Science Framework and can be accessed at https://osf.io/szn26/. Data were analyzed using R, Version 4.3.0 (R Core Team, 2023), the package mlpwr, Version 1.0.1 (Zimmer & Debelak, 2022a), and various other R packages listed in the online supplemental materials. All computations were performed on Science Cluster at the University of Zurich, a batchqueue cluster system service for research targeted to address largescale computational research.…”

The Influence of Mediation Approaches, Social Factors, and Psychological Factors on Family Dispute Resolution at the Ministry of Religious Affairs in West Java

“…Next, the structural model will examine the relationships and interactions among mediation approaches, social factors, psychological factors, and the resolution of family disputes [26]. Hypotheses derived from the literature review will be tested to determine the strength and significance of these relationships [27]. Bootstrapping will be employed to assess the robustness and stability of the results.…”

Simulation-based Design Optimization for Statistical Power: Utilizing Machine Learning

“…We implemented a tuning protocol for SVR that allows to find suitable hyperparameters for a variety of scenarios. Our approach is implemented in the mlpwr R package Zimmer, Henninger, et al, 2022).…”