Power gains by using external information in clinical trials are typically not possible when requiring strict type I error control

Abstract

In the era of precision medicine, novel designs are developed to deal with flexible clinical trials that incorporate many treatment strategies for multiple diseases in one trial setting. This situation often leads to small sample sizes in disease‐treatment combinations and has fostered the discussion about the benefits of borrowing of external or historical information for decision‐making in these trials. Several methods have been proposed that dynamically discount the amount of information borrowed from historical data based on the conformity between historical and current data. Specifically, Bayesian methods have been recommended and numerous investigations have been performed to characterize the properties of the various borrowing mechanisms with respect to the gain to be expected in the trials. However, there is common understanding that the risk of type I error inflation exists when information is borrowed and many simulation studies are carried out to quantify this effect. To add transparency to the debate, we show that if prior information is conditioned upon and a uniformly most powerful test exists, strict control of type I error implies that no power gain is possible under any mechanism of incorporation of prior information, including dynamic borrowing. The basis of the argument is to consider the test decision function as a function of the current data even when external information is included. We exemplify this finding in the case of a pediatric arm appended to an adult trial and dichotomous outcome for various methods of dynamic borrowing from adult information to the pediatric arm. In conclusion, if use of relevant external data is desired, the requirement of strict type I error control has to be replaced by more appropriate metrics.

Including non-concurrent control patients in the analysis of platform trials: is it worth it?

“…However there could be an inflation (or deflation) in the type one error rate and the marginal power of testing the corresponding hypothesis unless the positive (or negative) trend has a magnitude of less than 0.5% of the standard deviation (based on simulation results not presented here). This finding is similar to the finding of Kopp-Schneider et al [39] for the situation where external information is used in the inference of clinical trials: power gain is not possible when requiring a strict type one error rate control. When the recruitment to all arms do not finish simultaneously, the control data can be sepa-rated into before and after adding an arm and those after the initial treatment arms finish recruitment.…”

Incorporating historical two‐arm data in clinical trials with binary outcome: A practical approach

“…This can also be seen by considering the rejection regions presented for the clinical trial example ( Figure 5). Thus, in summary, our results are in line with the results of Kopp-Schneider et al 28 who argue that "borrowing of information cannot lead to an increased power while strictly controlling type I error." Our proposed procedure achieves this goal only in the two-arm borrowing and two-sided testing case by reducing the power to reveal a favorable treatment effect for the control group as compared to the treatment group, while still controlling the type I error at the prespecified significance level α.…”

Comments on “Incorporating historical two‐arm data in clinical trials with binary outcome: A practical approach” by Manuel Feißt, Johannes Krisam and Meinhard Kieser

“…Their approach corresponds to a Bayesian power prior approach with prespecified amount of borrowing. Their results seem to contradict the findings by Kopp-Schneider et al, 2 who showed that power gains from historical information are not possible when type I error rate should be controlled whenever a uniformly most powerful (UMP) test exists. Although not mentioned in the title nor in the abstract, the situation considered by Feißt and colleagues is the two-sided test setup for a two-arm comparison of a binary variable.…”