Availability of study protocols for randomized trials published in high-impact medical journals: A cross-sectional analysis

Johnson

et al. 2020

BMC Med

Background: Choosing or altering the planned statistical analysis approach after examination of trial data (often referred to as 'p-hacking') can bias the results of randomised trials. However, the extent of this issue in practice is currently unclear. We conducted a review of published randomised trials to evaluate how often a pre-specified analysis approach is publicly available, and how often the planned analysis is changed. Methods: A review of randomised trials published between January and April 2018 in six leading general medical journals. For each trial, we established whether a pre-specified analysis approach was publicly available in a protocol or statistical analysis plan and compared this to the trial publication. Results: Overall, 89 of 101 eligible trials (88%) had a publicly available pre-specified analysis approach. Only 22/89 trials (25%) had no unexplained discrepancies between the pre-specified and conducted analysis. Fifty-four trials (61%) had one or more unexplained discrepancies, and in 13 trials (15%), it was impossible to ascertain whether any unexplained discrepancies occurred due to incomplete reporting of the statistical methods. Unexplained discrepancies were most common for the analysis model (n = 31, 35%) and analysis population (n = 28, 31%), followed by the use of covariates (n = 23, 26%) and the approach for handling missing data (n = 16, 18%). Many protocols or statistical analysis plans were dated after the trial had begun, so earlier discrepancies may have been missed. Conclusions: Unexplained discrepancies in the statistical methods of randomised trials are common. Increased transparency is required for proper evaluation of results.

Section: Discussionmentioning

confidence: 98%

Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials

Johnson

et al. 2020

BMC Med

“…We note that transparency around the statistical methods used in clinical trials is increasing, and there are initiatives in place to further increase transparency (for example, those conducted by the UKCRC CTU network, https://www.ukcrc-ctu.org.uk/). However, there is still a long way to go; evidence shows that the statistical methods for the trial's primary outcome are often poorly specified in both trial protocols [5,26,27] and Statistical Analysis Plans [26]; that protocols and Statistical Analysis Plans are often not made publicly available, or are only done so after they may have already been modified during the course of the trial [26,27,29]; that undisclosed changes to the planned analysis approach are frequent [2,26,27]; and that reporting around data access and blinding status of statisticians is often poor [26,27], hampering the ability of readers to evaluate whether changes have been made based on unblinded trial data. Pre-SPEC can play a part, alongside other initiatives, to help increase transparency in clinical trials, and resolving some of the issues outlined above.…”

Section: Discussionmentioning

confidence: 99%

How to design a pre-specified statistical analysis approach to limit p-hacking in clinical trials: the Pre-SPEC framework

Kahan

2020

BMC Med

Results from clinical trials can be susceptible to bias if investigators choose their analysis approach after seeing trial data, as this can allow them to perform multiple analyses and then choose the method that provides the most favourable result (commonly referred to as 'p-hacking'). Pre-specification of the planned analysis approach is essential to help reduce such bias, as it ensures analytical methods are chosen in advance of seeing the trial data. For this reason, guidelines such as SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and ICH-E9 (International Conference for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use) require the statistical methods for a trial's primary outcome be pre-specified in the trial protocol. However, prespecification is only effective if done in a way that does not allow p-hacking. For example, investigators may prespecify a certain statistical method such as multiple imputation, but give little detail on how it will be implemented. Because there are many different ways to perform multiple imputation, this approach to pre-specification is ineffective, as it still allows investigators to analyse the data in different ways before deciding on a final approach. In this article, we describe a five-point framework (the Pre-SPEC framework) for designing a pre-specified analysis approach that does not allow p-hacking. This framework was designed based on the principles in the SPIRIT and ICH-E9 guidelines and is intended to be used in conjunction with these guidelines to help investigators design the statistical analysis strategy for the trial's primary outcome in the trial protocol.

“…Our results are broadly consistent with previous reviews. Spence et al (30) evaluated the availability of protocols and SAPs for trials published in high impact medical journals, and found similar rates of availability. However, the rates of discrepancies we found were generally lower than those previously reported (8, 10, 20, 21).…”

Section: Discussionmentioning

confidence: 99%

Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomized trials

Johnson

et al. 2020

Preprint

Background: Choosing or altering the planned statistical analysis approach after examination of trial data (often referred to as p-hacking) can bias results of randomized trials. However, the extent of this issue in practice is currently unclear. We conducted a review of published randomized trials to evaluate how often a pre-specified analysis approach is publicly available, and how often the planned analysis is changed. Methods: A review of randomised trials published between January and April 2018 in six leading general medical journals. For each trial we established whether a pre-specified analysis approach was publicly available in a protocol or statistical analysis plan, and compared this to the trial publication. Results: Overall, 89 of 101 eligible trials (88%) had a publicly available pre-specified analysis approach. Only 22/89 trials (25%) had no unexplained discrepancies between the pre-specified and conducted analysis. Fifty-four trials (61%) had one or more unexplained discrepancies, and in 13 trials (15%) it was impossible to ascertain whether any unexplained discrepancies occurred due to incomplete reporting of the statistical methods. Unexplained discrepancies were most common for the analysis model (n=31, 35%) and analysis population (n=28, 31%), followed by the use of covariates (n=23, 26%) and the approach for handling missing data (n=16, 18%). Many protocols or statistical analysis plans were dated after the trial had begun, so earlier discrepancies may have been missed. Conclusions: Unexplained discrepancies in the statistical methods of randomized trials are common. Increased transparency is required for proper evaluation of results.