A framework for extending trial design to facilitate missing data sensitivity analyses

Abstract

Background: Missing data are an inevitable challenge in Randomised Controlled Trials (RCTs), particularly those with Patient Reported Outcome Measures. Methodological guidance suggests that to avoid incorrect conclusions, studies should undertake sensitivity analyses which recognise that data may be 'missing not at random' (MNAR). A recommended approach is to elicit expert opinion about the likely outcome differences for those with missing versus observed data. However, few published trials plan and undertake these elicitation exercises, and so lack the external information required for these sensitivity analyses. The aim of this paper is to provide a framework that anticipates and allows for MNAR data in the design and analysis of clinical trials. Methods: We developed a framework for performing and using expert elicitation to frame sensitivity analysis in RCTs with missing outcome data. The framework includes the following steps: first defining the scope of the elicitation exercise, second developing the elicitation tool, third eliciting expert opinion about the missing outcomes, fourth evaluating the elicitation results, and fifth analysing the trial data. We provide guidance on key practical challenges that arise when adopting this approach in trials: the criteria for identifying relevant experts, the outcome scale for presenting data to experts, the appropriate representation of expert opinion, and the evaluation of the elicitation results. The framework was developed within the POPPI trial, which investigated whether a preventive, complex psychological intervention, commenced early in ICU, would reduce the development of patient-reported post-traumatic stress disorder symptom severity, and improve health-related quality of life. We illustrate the key aspects of the proposed framework using the POPPI trial. Results: For the POPPI trial, 113 experts were identified with potentially suitable knowledge and asked to participate in the elicitation exercise. The 113 experts provided 59 usable elicitation questionnaires. The sensitivity analysis found that the results from the primary analysis were robust to alternative MNAR mechanisms. Conclusions: Future studies can adopt this framework to embed expert elicitation within the design of clinical trials. This will provide the information required for MNAR sensitivity analyses that examine the robustness of the trial conclusions to alternative, but realistic assumptions about the missing data.

Insights From Survival Analyses During 12 Years of Anti–Vascular Endothelial Growth Factor Therapy for Neovascular Age-Related Macular Degeneration

“…In a review of RCTs published between July and December 2013, the median proportion of participants with a missing outcome was 9% . Missing observations in RCTs are assumed to occur randomly, and the appropriate statistical methods to analyze outcomes are chosen; that is, it is generally accepted that multiple imputation models for missing data and the use of mixed-effects models provide better outcome estimates than using only observed data or last observation carried forward in clinical trials . Such approaches typically cannot be used in retrospective studies of clinical practice data, or so-called real-world studies, which also usually have a high rate of missing data (17%-34% at 1 year); the circumstances of nonobservation are rarely described, and the underlying reasons cannot be assumed to be random .…”

Determining the sample size for a cluster-randomised trial using knowledge elicitation: Bayesian hierarchical modelling of the intracluster correlation coefficient

“…As an extension, strategies for evaluating the elicitation exercise (including ranking reviewers, agreement and coherence checks, calibration) 30 can be embedded within the elicitation process, together with examining sensitivity of the conclusions to the used models. 44 A more advanced approach would be to set up calibration questions, where experts are asked questions where the truth is known. 45 The choice of the reviewers in this study was a convenience purposive expert sample, and it could be improved in further practical implementation of the method.…”

Reduced exposure to vasopressors through permissive hypotension to reduce mortality in critically ill people aged 65 and over: the 65 RCT

“…For example, we may expect that patients who are in a relatively good state of health may be more likely to complete and return the HRQoL questionnaire, and this would mean that these outcome data may be 'missing not at random' (MNAR). The steps in the expert elicitation framework 52 were followed for this additional analysis. These steps included (1) scoping a 65 trial-specific elicitation exercise, (2) development of an elicitation tool (including questions about the HRQoL outcomes), (3) eliciting expert opinion, (4) evaluating the elicitation results and (5) carrying out the sensitivity analysis, incorporating the elicited expert information.…”

“…Following the approach discussed in Mason et al, 52 we used a combination of pooled and individual priors to fully explore the sensitivity of the trial results to a range of expert opinion. The individual priors were selected from the very high-confidence subgroup.…”