Calculating the Expected Value of Sample Information in Practice: Considerations from 3 Case Studies

Abstract

Investing efficiently in future research to improve policy decisions is an important goal. Expected Value of Sample Information (EVSI) can be used to select the specific design and sample size of a proposed study by assessing the benefit of a range of different studies. Estimating EVSI with the standard nested Monte Carlo algorithm has a notoriously high computational burden, especially when using a complex decision model or when optimizing over study sample sizes and designs. Therefore, a number of more efficient EVSI approximation methods have been developed. However, these approximation methods have not been compared and therefore their relative advantages and disadvantages are not clear.A consortium of EVSI researchers, including the developers of several approximation methods, compared four EVSI methods using three previously published health economic models. The examples were chosen to represent a range of real-world contexts, including situations with multiple study outcomes, missing data, and data from an observational rather than a randomized study. The computational speed and accuracy of each method were compared, and the relative advantages and implementation challenges of the methods were highlighted.In each example, the approximation methods took minutes or hours to achieve reasonably accurate EVSI estimates, whereas the traditional Monte Carlo method took weeks. Specific methods are particularly suited to problems where we wish to compare multiple proposed sample sizes, when the proposed sample size is large, or when the health economic model is computationally expensive. All the evaluated methods gave estimates similar to those given by traditional Monte Carlo, suggesting that EVSI can now be efficiently computed with confidence in realistic examples.

Building a trusted framework for uncertainty assessment in rare diseases: suggestions for improvement (Response to “TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases”)

“…First, consideration has to be given to potentially appropriate risk-sharing arrangements. Risk-sharing arrangements can then be assessed quantitatively where possible, for example using the HTA risk analysis framework [17,26], which requires value of information analysis [27][28][29][30]. Further research could inform a more qualitative approach to identifying research targets, study designs and negotiating an acceptable agreement with manufacturers to ensure appropriate pricing.…”

How can health economics be used in the design and analysis of adaptive clinical trials? A qualitative analysis

“…• The Professional Society for Health Economics and Outcomes Research (ISPOR) Value of Information Analysis for Research Decisions task force, which aims to develop good practice guidance for using methods of value of information analysis to inform both technology reimbursement decisions and research prioritisation decisions [36] • Collaborative Network for Value of Information (ConVOI) group, an international network working to improve the calculation, adoption and application of value of information methods in clinical and public health research [37,38].…”

Computing the Expected Value of Sample Information Efficiently: Practical Guidance and Recommendations for Four Model-Based Methods

“…Thus, EVSI calculations will soon be required from analysts. Until recently, EVSI calculations were extremely computationally expensive, potentially taking weeks or months, 8 because they required nested simulation methods. 9 Thus most VOI analyses were restricted to computing the expected value of perfect information (EVPI) or the expected value of partial perfect information (EVPPI), for which several efficient methods are available.…”

“…Recently, the accuracy and computational time of these methods were compared using 3 different case studies, and researchers concluded that none of the methods were systematically superior in terms of accuracy, precision, or computational time. 8 Still, these methods differ in their approach, requiring different expertise and skills. Nevertheless, because there is no structured comparison of the practical steps required to use them, it is challenging for analysts to determine which method is appropriate for their situation and expertise.…”