Conducting Value for Money Analyses for Non-randomised Interventional Studies Including Service Evaluations: An Educational Review with Recommendations

Abstract

This article provides an educational review covering the consideration of conducting 'value for money' analyses as part of non-randomised study designs including service evaluations. These evaluations represent a vehicle for producing evidence such as value for money of a care intervention or service delivery model. Decision makers including charities and local and national governing bodies often rely on evidence from non-randomised data and service evaluations to inform their resource allocation decision-making. However, as randomised data obtained from randomised controlled trials are considered the 'gold standard' for assessing causation, the use of this alternative vehicle for producing an evidence base requires careful consideration. We refer to value for money analyses, but reflect on methods associated with economic evaluations as a form of analysis used to inform resource allocation decision-making alongside a finite budget. Not all forms of value for money analysis are considered a full economic evaluation with implications for the information provided to decision makers. The type of value for money analysis to be conducted requires considerations such as the outcome(s) of interest, study design, statistical methods to control for confounding and bias, and how to quantify and describe uncertainty and opportunity costs to decision makers in any resulting value for money estimates. Service evaluations as vehicles for producing evidence present different challenges to analysts than what is commonly associated with research, randomised controlled trials and health technology appraisals, requiring specific study design and analytic considerations. This educational review describes and discusses these considerations, as overlooking them could affect the information provided to decision makers who may make an 'ill-informed' decision based on 'poor' or 'inaccurate' information with long-term implications. We make direct comparisons between randomised controlled trials relative to non-randomised data as vehicles for assessing causation; given 'gold standard' randomised controlled trials have limitations. Although we use UK-based decision makers as examples, we reflect on the needs of decision makers internationally for evidence-based decision-making specific to resource allocation. We make recommendations based on the experiences of the authors in the UK, reflecting on the wide variety of methods available, used as documented in the empirical literature. These methods may not have been fully considered relevant to non-randomised study designs and/or service evaluations, but could improve and aid the analysis conducted to inform the relevant value for money decision problem.

Cost-Utility Analysis of Major System Change in Specialist Cancer Surgery in London, England, Using Linked Patient-Level Electronic Health Records and Difference-in-Differences Analysis

“…We did not have explicit information on cancer progression status over the 10-year period, except indirectly in the ‘not healthy’ group during the decision tree, and the use of a single ‘alive’ health state in the Markov section of the model is a limitation of the analysis. Randomised controlled trials are the gold standard to control for potential confounders, but they are rarely possible in service change evaluations, so, we used difference-in-differences methodology, adjusting for available baseline patient and disease characteristics recommended by the clinical authors and other clinical colleagues, to control for other contemporaneous changes that might have taken place [ 56 ]. Sites in GM were excluded from the ROE group to accommodate the parallel trends assumption in difference-in-differences analysis, and this was acceptable because the main statistical analysis performed pre-trends tests to ascertain whether any difference in the linear trends in case-mix adjusted outcomes was apparent, when comparing values in the LC region with those in the ROE before the centralisations.…”

Harmonization issues in unit costing of service use for multi-country, multi-sectoral health economic evaluations: a scoping review

“…Both, extensive sensitivity analysis or transparent overhead calculation as one harmonization aspect for unit costs would be potential ways forward. There are, however, also broader costing issues [ 45 ], such as questions about the inclusion of future medical costs [ 75 ], the choice of the discount rate [ 40 ], and the choice of the analytical study perspective [ 76 ] that may introduce systematic differences in cost estimates [ 77 , 78 ]. Secondly, where newly developed harmonization strategies are not fully in line with existing national EE guidelines, especially those relevant for reimbursement decisions, their implementation most likely will face resistance.…”

Economic models of community-based falls prevention: a systematic review with subsequent commissioning and methodological recommendations

“…This presented difficulties in certain areas (e.g., whether Markov model incorporated tunnel states); contacting the study authors for enquiry would have reduced ambiguity. Secondly, in several ROI analyses (e.g., [56,59,72]), it was unclear whether the analyses constituted full comparative economic evaluations or non-comparative service evaluations (i.e., a partial economic evaluation) [119]. Clearer description of the evaluation aim and detailed parameterisation of the comparator scenario would facilitate future distinctions.…”