Subjects, Trials, and Levels: Statistical Power in Conjoint Experiments

Abstract

Conjoint analysis is an experimental technique that has become quite popular to understand people's decisions in multi-dimensional decision-making processes. Despite the importance of power analysis for experimental techniques, current literature has largely disregarded statistical power considerations when designing conjoint experiments. The main goal of this article is to provide researchers and practitioners with a practical tool to calculate the statistical power of conjoint experiments. To this end, we first conducted an extensive literature review to understand how conjoint experiments are designed and gauge the plausible effect sizes discovered in the literature. Second, we formulate a data generating model that is sufficiently flexible to accommodate a wide range of conjoint designs and hypothesized effects. Third, we present the results of an extensive series of simulation experiments based on the previously formulated data generation process. Our results show that---even with relatively large sample size and the number of trials---conjoint experiments are not suited to draw inferences for experiments with large numbers of experimental conditions and relatively small effect sizes. Specifically, Type S and Type M errors are especially pronounced for experimental designs with relatively small effective sample sizes (< 3000) or a high number of levels (> 15) that find small but statistically significant effects (< 0.03). The proposed online tool based on the simulation results can be used by researchers to perform power analysis of their designs and hence achieve adequate design for future conjoint experiments.

Using Conjoint Experiments to Study Preferences in Multidimensional Choice Contexts

“…These categories needed to be substantively informative, feasible, and concise to minimize the cognitive burden on respondents. Besides this, the number of levels (the attribute with the greatest number of levels) is decisive in deriving the desired sample size (for power analysis in conjoint experiments, see Schuessler &Freitag, 2020 andStefanelli &Lukac, 2020). For that reason, the general advice is to minimize the number of levels, but of course, not at the expense of compromising the validity of the study.…”

“…Conduct a power analysis. Derive a desired sample size for detecting effect sizes of interest based on power analysis tools created by Schuessler and Freitag (2020) and Stefanelli & Lukac (2020).…”

The lure of populism: a conjoint experiment examining the interplay between demand and supply side factors

“…However, a recent contribution makes it possible to gauge the power of conjoint designs (Lukac and Stefanelli, 2020). Through simulations, the shiny app makes it possible to gauge the power of the conjoint experiment when varying key parameters such as sample size, number of comparisons, effect sizes (Average Marginal Component Effects) and attribute levels (Stefanelli and Lukac, 2020). We used this approach to examine what effect sizes were detectable with our experiment.…”

Power Analysis for Conjoint Experiments

“…4 If little or weak prior information is available, broad meta-averages (c.f. Stefanelli and Lukac, 2020) provide at least rough bounds.…”

“…Existing power calculations for conjoint studies based on simulation (Gall, 2020;Stefanelli and Lukac, 2020) are limited to non-interactive quantities, rely on causal models that make parametric and distributional assumptions, and often need multiple user inputs that are not reported in standard empirical analyses. Moreover, the computational burden -even with parallel computing -is not trivial, rendering integration into standard research practice difficult.…”