The population-wise error rate for clinical trials with overlapping populations

Abstract

We introduce a new multiple type I error criterion for clinical trials with multiple, overlapping populations. Such trials are of interest in precision medicine where the goal is to develop treatments that are targeted to specific sub-populations defined by genetic and/or clinical biomarkers. The new criterion is based on the observation that not all type I errors are relevant to all patients in the overall population. If disjoint sub-populations are considered, no multiplicity adjustment appears necessary, since a claim in one sub-population does not affect patients in the other ones. For intersecting sub-populations we suggest to control the average multiple type I error rate, i.e. the probability that a randomly selected patient will be exposed to an inefficient treatment. We call this the population-wise error rate, exemplify it by a number of examples and illustrate how to control it with an adjustment of critical boundaries or adjusted [Formula: see text]-values. We furthermore define corresponding simultaneous confidence intervals. We finally illustrate the power gain achieved by passing from family-wise to population-wise error rate control with two simple examples and a recently suggested multiple-testing approach for umbrella trials.

Rejoinder to Discussions on “Optimal Test Procedures for Multiple Hypotheses Controlling the Familywise Expected Loss”

“…It is remarkable that the works by Brannath et al. (2023) and Maurer et al. (2023) in extending conventional error rates were triggered by similar applications.…”

“…We particularly enjoyed learning from Brannath (2023) how the FWEL specializes to the populationwise error rate (PWER) recently introduced by Brannath et al. (2023) through a loss function that combines additive and binary loss functions in a novel way. For the disjoint subgroup analysis application discussed in our paper, the PWER reduces to the FWEL with an additive loss function and losses $$\lambda _i$$ being proportional to the prevalence $$\pi _i$$ of subgroup i .…”

“…The result from Brannath (2023) also shows how weights arise "naturally" in subgroup analyses through the prevalences 𝜋 𝑖 . It is remarkable that the works by Brannath et al (2023) and Maurer et al (2023) in extending conventional error rates were triggered by similar applications. Agreeing on the costs (and gains) in other settings than subgroup analyses may be more challenging, although we believe that the FWEL remains applicable.…”

Optimal test Procedures for Multiple Hypotheses Controlling the Familywise Expected Loss

“…If, however, only the hypothesis 𝐻 1 for the first subgroup is rejected, the new treatment may be approved only for that subgroup 𝑆 1 . The risk of a false positive decision for that subgroup is therefore controlled by testing 𝐻 1 at level 𝛼, as also noticed by Brannath et al (2023). Since the same argument also holds for the other subgroup 𝑆 2 , it then seems reasonable to test each individual hypothesis 𝐻 1 and 𝐻 2 at level 𝛼, although the FWER can be almost 2𝛼.…”

Discussion on “Optimal Test Procedures for Multiple Hypotheses Controlling the Familywise Expected Loss” by Willi Maurer, Frank Bretz, and Xiaolei Xun

“…They suggest that the relative size of the sub‐population $$|S_k|$$ can serve as predetermined weights in the loss function as argued in Brannath et al. (2021). In the proposed framework, a question remains as to the assignment of weights in the gain function: following the argument of ‘error per person’ in the loss, so should the gains be weighted as ‘gain per person’ leading to weights proportional to $$|S_k|$$.…”