The Estimating effectiveness from efficacy taxonomy (EFFECT): A tool to estimate the real-world impact of health interventions

Abstract

To develop and pilot test a taxonomy that empirically estimates health intervention effectiveness from efficacy data. Methods: We developed a taxonomy to score health interventions across 11 items on a scale from 0-100. The taxonomy was pilot-tested in efficacy and effectiveness diabetes prevention studies identified in two separate systematic reviews; here, the face validity, interrater reliability and factor structure of the taxonomy were established. Random effects meta-analyses were used to obtain weight loss and diabetes incidence pooled effects across studies. These effects and taxonomy scores were used to down calibrate efficacy estimates to effectiveness estimates as follows: Efficacy effect*[Efficacy score/highest possible score]. Results: We scored 82 effectiveness lifestyle modification studies (mean score 49.2), 32 efficacy lifestyle modification studies (mean score 69.8) and 20 efficacy studies testing medications (mean score 77.4). The taxonomy had face validity and good inter-rater reliability (ICC = 0.9 [0.87, 0.93]). The between-groups down calibrated weight loss estimate was similar to that observed in the effectiveness meta-analysis (1.7 and 1.8 kg, respectively). The down calibrated diabetes relative risk reduction was also similar to that observed in the effectiveness meta-analysis (30.6% over 2.7 years and 29% over 2 years, respectively). Conclusions: The taxonomy is a promising tool to estimate the real-world impact of health interventions.