Optimal test Procedures for Multiple Hypotheses Controlling the Familywise Expected Loss

Abstract

We consider the problem of testing multiple null hypotheses, where a decision to reject or retain must be made for each one and embedding incorrect decisions into a real‐life context may inflict different losses. We argue that traditional methods controlling the Type I error rate may be too restrictive in this situation and that the standard familywise error rate may not be appropriate. Using a decision‐theoretic approach, we define suitable loss functions for a given decision rule, where incorrect decisions can be treated unequally by assigning different loss values. Taking expectation with respect to the sampling distribution of the data allows us to control the familywise expected loss instead of the conventional familywise error rate. Different loss functions can be adopted, and we search for decision rules that satisfy certain optimality criteria within a broad class of decision rules for which the expected loss is bounded by a fixed threshold under any parameter configuration. We illustrate the methods with the problem of establishing efficacy of a new medicinal treatment in non‐overlapping subgroups of patients.

Rejoinder to Discussions on “Optimal Test Procedures for Multiple Hypotheses Controlling the Familywise Expected Loss”

“…The result from Brannath (2023) also shows how weights arise "naturally" in subgroup analyses through the prevalences 𝜋 𝑖 . It is remarkable that the works by Brannath et al (2023) and Maurer et al (2023) in extending conventional error rates were triggered by similar applications. Agreeing on the costs (and gains) in other settings than subgroup analyses may be more challenging, although we believe that the FWEL remains applicable.…”

“…In Maurer et al (2023), we introduced the familywise expected loss (FWEL) and focused on additive and binary step loss functions. We pointed out that the FWEL specializes to the conventional familywise error rate (FWER) when using binary step loss functions.…”

“…Furthermore, the FWEL specializes to the per comparison error rate (PCER) and the per family error rate (PFER) when using additive step loss functions for specific choices of the losses 𝜆 𝑖 , as nicely illustrated by Benjamini et al (2023). They argued against using the PCER and referred to the decision rule in the lower left panel of Figure 2 in Maurer et al (2023) for potential consequences in terms of rejection thresholds becoming larger than 𝛼 if the tests had to be performed separately. These consequences may be reasonable, however, as long as the implications of an erroneous rejection (i.e., the costs) are well understood.…”

“…Using level-𝛼 tests in each of the two regions essentially means using a conservative testing approach. Instead, thresholds like those shown in Table 2 of Maurer et al (2023) can be used. In practice, however, one would have to consider similarities to off-label use and the motivation for using sub-additive loss functions.…”