Economic Analysis of a Multi-Site Prevention Program: Assessment of Program Costs and Characterizing Site-level Variability

Abstract

Programmatic cost analyses of preventive interventions commonly have a number of methodological difficulties. To determine the mean total costs and properly characterize variability, one often has to deal with small sample sizes, skewed distributions, and especially missing data. Standard approaches for dealing with missing data such as multiple imputation may suffer from a small sample size, a lack of appropriate covariates, or too few details around the method used to handle the missing data. In this study, we estimate total programmatic costs for a prevention trial evaluating the Strong African American Families-Teen program. This intervention focuses on the prevention of substance abuse and risky sexual behavior. To account for missing data in the assessment of programmatic costs we compare multiple imputation to probabilistic sensitivity analysis. The latter approach uses collected cost data to create a distribution around each input parameter. We found that with the multiple imputation approach, the mean (95% confidence interval) incremental difference was $2149 ($397, $3901). With the probabilistic sensitivity analysis approach, the incremental difference was $2583 ($778, $4346). Although the true cost of the program is unknown, probabilistic sensitivity analysis may be a more viable alternative for capturing variability in estimates of programmatic costs when dealing with missing data, particularly with small sample sizes and the lack of strong predictor variables. Further, the larger standard errors produced by the probabilistic sensitivity analysis method may signal its ability to capture more of the variability in the data, thus better informing policymakers on the potentially true cost of the intervention.

Understanding cost data collection tools to improve economic evaluations of health interventions

“…But many preventive interventions are set in community-based settings and might not have the data infrastructure found in healthcare settings, which could limit the feasibility of using existing data systems to track resource use. Moreover, many prevention interventions are delivered in multiple sites with varying contexts that can influence costs,29 which further necessitates a standard primary data collection strategy. Direct observation could produce very accurate and precise data as trained observers watch the intervention processes and consistently record the resources used.…”

“…While not as high cost as direct observation, activity logs still come with the additional research burden as staff must complete the logs during their work day, where they may already be working at capacity, and could require training for the logs’ use. And because the logs rely on multiple staff to input reliable data, results depend on individuals’ agreement to participate in the data collection and their level of buy-in and effort, which can lead to missing data 29…”

“…But in contexts where this might not be feasible for all cost categories and prioritisations need to be made (eg, limited research budget, time, site buy-in), some costs might be more important to track in detail than others. For example, interventions to prevent and manage chronic conditions tend to be heavily labour intensive, and thus labour cost is typically a key category for understanding programme costs 29 30. Community-based interventions employ labour from the setting in which it is delivered (eg, school, community centre) and staff members may have multiple job duties, related and unrelated to the intervention, meaning a detailed understanding of how their time is spent is essential to allocate an appropriate amount of their time to the intervention and avoid overestimating its costs.…”

Benefit-Cost Analysis of Prevention and Intervention Programs for Youth and Young Adults: Introduction to the Special Issue

“…Earlier prevention and intervention cost analyses have helped illustrate a variety of issues that need to be considered in thorough, high-quality cost analyses. These include assessing how intervention activities and associated costs vary over the life of a program (Crowley, Jones, Greenberg, Feinberg & Spoth, 2012), the costs of training and technical assistance that support program efficacy (Kuklinski, Briney, Hawkins & Catalano, 2012), and site-level variability in costs in multisite trials (Corso et al., 2013).…”