How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework

Abstract

Results from clinical trials can be susceptible to bias if investigators choose their analysis approach after seeing trial data, as this can allow them to perform multiple analyses and then choose the method that provides the most favourable result (commonly referred to as 'p-hacking'). Pre-specification of the planned analysis approach is essential to help reduce such bias, as it ensures analytical methods are chosen in advance of seeing the trial data. For this reason, guidelines such as SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and ICH-E9 (International Conference for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use) require the statistical methods for a trial's primary outcome be pre-specified in the trial protocol. However, prespecification is only effective if done in a way that does not allow p-hacking. For example, investigators may prespecify a certain statistical method such as multiple imputation, but give little detail on how it will be implemented. Because there are many different ways to perform multiple imputation, this approach to pre-specification is ineffective, as it still allows investigators to analyse the data in different ways before deciding on a final approach. In this article, we describe a five-point framework (the Pre-SPEC framework) for designing a pre-specified analysis approach that does not allow p-hacking. This framework was designed based on the principles in the SPIRIT and ICH-E9 guidelines and is intended to be used in conjunction with these guidelines to help investigators design the statistical analysis strategy for the trial's primary outcome in the trial protocol.

Public availability and adherence to prespecified statistical analysis approaches was low in published randomized trials

“…Of the trials that did have a pre-specified analysis approach available, we found that almost all of them (92%) had unexplained discrepancies. These typically involved making undisclosed changes to the pre-planned methods, or pre-specifying the methods in a manner which allowed p-hacking (11). Because of poor reporting around blinding of statisticians and access to data, it was often difficult to ascertain whether changes were made before or after access to the trial data.…”

“…This would ensure that most trials could be assessed for inappropriate changes to the analysis approach. Next, authors could ensure they do not prespecify their analysis approach in a way which allows p-hacking; the Pre-SPEC framework, which has recently become available, provides guidance on this exact issue (11). Finally, editors could require that authors report any changes to the pre-specified methods, as well as disclose the blinding status of investigators at the time of each change (authors could also do so voluntarily).…”

“…The second discrepancy was termed an 'addition', which occurred if essential details about how the analysis would be implemented were missing from the pre-specified analysis approach. 'Additions' were considered problematic as it meant the analysis approach was pre-specified in a manner which still allowed investigators to decide on some details of how the analysis would be implemented after seeing the trial data, which could allow p-hacking (11). Additions occurred if the pre-specified analysis approach (1) contained insufficient information about the proposed analysis; or (2) allowed the investigators to subjectively choose between multiple different potential analyses.…”

“…The statistical methods used to analyse a randomised trial can affect the results; for instance, excluding different participants or using different statistical models can change the size of the estimated treatment effect or p-value (1)(2)(3)(4)(5)(6)(7)(8)(9)(10)(11)(12)(13)(14). Selective reporting of results is problematic in randomised trials (4,6,7,13,15), and, choosing or modifying the planned analysis approach after seeing the trial data to obtain a more favourable result can introduce bias; this is commonly referred to as "p-hacking" (1-5, 11, 12, 15).…”

The Effects of Dietary Protein Supplementation on Exercise-Induced Inflammation and Oxidative Stress: A Systematic Review of Human Trials

“…Akin to several previous systematic reviews with nutritional interventions [ 58 , 90 ], most of the included studies were deemed to have some concerns, because there was insufficient information on the randomization and allocation concealment procedures, as well as missing information about a prespecified analysis plan in each study. The former is potentially problematic because it can result in selection bias [ 91 ] and the latter because, without evidence of a prespecified analysis plan, it is difficult to know whether any markers have not been reported or if the analysis was selected post hoc to fit the hypothesis [ 92 ].…”