The Burden of the “False‐Negatives” in Clinical Development: Analyses of Current and Alternative Scenarios and Corrective Measures

Abstract

The "false-negatives" of clinical development are the effective treatments wrongly determined ineffective. Statistical errors leading to "false-negatives" are larger than those leading to "false-positives," especially in typically underpowered early-phase trials. In addition, "false-negatives" are usually eliminated from further testing, thereby limiting the information available on them. We simulated the impact of early-phase power on economic productivity in three developmental scenarios. Scenario 1, representing the current status quo, assumed 50% statistical power at phase II and 90% at phase III. Scenario 2 assumed increased power (80%), and Scenario 3, increased stringency of alpha (1%) at phase II. Scenario 2 led, on average, to a 60.4% increase in productivity and 52.4% increase in profit. Scenario 3 had no meaningful advantages. Our results suggest that additional costs incurred by increasing the power of phase II studies are offset by the increase in productivity. We discuss the implications of our results and propose corrective measures. Clin Transl Sci (2017) 10, 470-479; doi:10.1111/cts.12478; published online on 4 July 2017. Study Highlights WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?✔ Early-phase clinical development studies are usually underpowered, with little knowledge about the extent, magnitude, and economic impact of the consequent "falsenegatives." Only one brief previous report, using different methodology, has studied the topic. 48 WHAT QUESTION DID THIS STUDY ADDRESS?✔ Our simulations aimed to study the impact of statistical error thresholds on clinical development productivity. WHAT THIS STUDY ADDS TO OUR KNOWLEDGE ✔ Underpowered phase II studies result in unacceptably high rates of "false-negatives." The burden of "falsenegatives" on clinical development productivity is potentially enormous, leading to loss of effective treatments and associated commercial profits. Increasing the power of early-phase trials is worth the investment in larger sample sizes. HOW THIS MIGHT CHANGE CLINICAL PHARMACOL-OGY OR TRANSLATIONAL SCIENCE ✔ Increasing power of early-phase clinical trials could improve productivity of drug development with increased profits due to reduction in frequency of "false-negatives" compensating the costs of larger sample-sized studies.Clinical development is increasingly a complex, risky, lengthy, failure-prone, and costly process with considerable healthcare benefits and commercial profits at stake.1-4 Contributing to the costs and delays are statistical errors that lead to "false-positive" and "false-negative" results. The "falsepositives" are the treatments that appear promising but in fact are not. These errors can lead to expensive followup testing, exposure to unnecessary risks and ineffective treatments, and potentially costly delays in development of promising back-up treatments. The "false-negatives" are the effective treatments wrongly eliminated, leading to missed healthcare and economic opportunities and are the subject of our investigation. While the "falseness" o...

How technical progress reshaped behavioral neuroendocrinology during the last 50 years… and some methodological remarks

“…Multiple statistical approaches can quickly be used to analyze a same set of data, which generates the temptation of p-hacking, i.e., manipulation of data in a way that produces a desired (low) p value (Raj et al, 2018;Ulrich and Miller, 2015). The easy access to the entire literature has also generated a fascination for rankings and for publication in high impact factor journals of perfect, often underpowered, stories (Burt et al, 2017;Button et al, 2013) that sometimes represent selections of positive data ignoring the many failed attempts, which creates a false impression of the current state of knowledge (Brembs et al, 2013).…”

Prospectively validated augmented intelligence for disease-agnostic predictions of clinical success for novel therapeutics

“…Given that the reliability of ground-truth determination is capped by the 85.1% theoretical maximal prediction accuracy of drug clinical efficacy by human clinical trials 34 , we should refrain from directly labeling missed predictions as false without further consideration of subsequent developments. In particular, we should pay attention to missed failure predictions for phase 2b trials that are less statistically powered than phase 3 trials.…”

Sample Adequacy Control (SAC) Lowers False Negatives and Increases the Quality of Screening: Introduction of “Non-Competitive” SAC for qPCR Assays

“…Impact of false negatives on infection control: Poor assay sensitivity is directly proportional to the number of false negative results [ 7 ]. However, even assays with the most optimal analytical performance can yield FN results: e.g., when sampling is inadequate (below the minimal quantity of biological material collected), or the integrity of the sample is compromised during transportation and/or storage, when the assay is inhibited, or when there is any other pre-analytical manipulation compromising the accuracy of reported results [ 2 , 8 , 9 , 10 , 11 ]. Although these can be largely mitigated by streamlined and effective testing pipelines, the need for sample adequacy controls become critically essential, as testing volumes increase, when sample types become diverse and when self-sampling strategies become incorporated into testing algorithms.…”

“…Initially, these were limited to nasopharyngeal swabs, but now have been extended to saliva, mouth gargles, nasal, oral and rectal swabs, as well as self-collected samples [ 10 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ]. The relative frequency of FN samples is generally high (2–38%) and dependent on multiple variables [ 7 , 9 , 10 ]. FN screening results for COVID-19 test have detrimental social, economic and health impact, by offsetting the effectiveness of screening efforts and prolonging the pandemic.…”