The statistical interpretation of pilot trials: should significance thresholds be reconsidered?

Abstract

BackgroundIn an evaluation of a new health technology, a pilot trial may be undertaken prior to a trial that makes a definitive assessment of benefit. The objective of pilot studies is to provide sufficient evidence that a larger definitive trial can be undertaken and, at times, to provide a preliminary assessment of benefit.MethodsWe describe significance thresholds, confidence intervals and surrogate markers in the context of pilot studies and how Bayesian methods can be used in pilot trials. We use a worked example to illustrate the issues raised.ResultsWe show how significance levels other than the traditional 5% should be considered to provide preliminary evidence for efficacy and how estimation and confidence intervals should be the focus to provide an estimated range of possible treatment effects. We also illustrate how Bayesian methods could also assist in the early assessment of a health technology.ConclusionsWe recommend that in pilot trials the focus should be on descriptive statistics and estimation, using confidence intervals, rather than formal hypothesis testing and that confidence intervals other than 95% confidence intervals, such as 85% or 75%, be used for the estimation. The confidence interval should then be interpreted with regards to the minimum clinically important difference. We also recommend that Bayesian methods be used to assist in the interpretation of pilot trials. Surrogate endpoints can also be used in pilot trials but they must reliably predict the overall effect on the clinical outcome.

Mindfulness-based exposure and response prevention for obsessive compulsive disorder: Findings from a pilot randomised controlled trial

“…In pilot trials the guidance is only to report descriptive statistics (Lee, Whitehead, Jacques, & Julious, 2014). We report descriptive statistics for all participant characteristics and clinical outcomes in the form of counts, proportions, means, standard deviations and ranges as appropriate.…”

Psychomotor Therapy as an Additive Intervention for Violent Forensic Psychiatric Inpatients: A Pilot Study

“…Because the present study only included a relatively small number of patients, which in addition were divided into two treatment groups, its statistical power was rather limited. Therefore, treatment effects were evaluated by the minimum clinically important difference (MCID) method (Jaeschke, Singer, & Guyatt, 1989), which provides a threshold for the smallest difference that can be regarded as a clinically meaningful change as compared to the pretreatment assessment (Lee, Whitehead, Jacques, & Julious, 2014).…”

Brief Interpersonal Psychotherapy for depression during pregnancy in a low-income population: A randomized controlled trial

“…As this study was designed to be a pilot feasibility study and does not have sufficient power to detect statistical significance between groups, we instead report descriptive statistics, and effect sizes and confidence intervals for our three clinical outcome measures (EDS, brief STAI, and SSQR)(Lee et al, 2014; Thabane et al, 2010). …”