An Evaluation Service for Digital Public Health Interventions: User-Centered Design Approach (Preprint)

Abstract

BACKGROUND Digital health interventions have potential to improve public health by combining effective intervention and population reach. However, what biomedical researchers and digital developers consider an effective intervention differs, thereby creating an ongoing challenge to integrate their respective approaches when evaluating digital health interventions. OBJECTIVE Public Health England set out to operationalise an evaluation framework that combines biomedical and digital approaches, and demonstrates the impact, cost-effectiveness and benefit of digital health interventions to public health. METHODS A multidisciplinary project team, composed of service designers, academics and public health professionals, employed user-centred design methods such as qualitative research, engagement with end-users and stakeholders, and iterative learning. An iterative approach enabled the team to sequentially define the problem, understand user needs, identify opportunity areas, develop concepts, test prototypes, and plan service implementation. Outputs were critiqued by stakeholders, system leaders and a Working Group. RESULTS Semi-structured interviews (N=15) identified 26 themes and 82 user needs, expressed as 46 Jobs To Be Done, which were then validated across the journey of evaluation design for a digital health intervention. Seven essential concepts for evaluating digital health interventions were identified: (i) Evaluation Thinking; (ii) Evaluation Canvas; (iii) Contract Assistant; (iv) Testing Toolkit; (v) Development History; (vi) Data Hub, and (vii) Publish Health Outcomes. Three concepts were prioritised for further testing and development and subsequently refined into the proposed Public Health England Evaluation Service for public health digital health interventions. Testing with Public Health England's Couch-to-5K App digital team confirmed the viability, desirability and feasibility of both the evaluation approach and the Evaluation Service. CONCLUSIONS An iterative, user-centred design approach enabled Public Health England to combine the strengths of academic and biomedical disciplines alongside the expertise of non-academic and/or digital developers for evaluating digital health interventions. Design-led methodologies can add value in a public health setting. The subsequent service, now known as Evaluating Digital Health Products, is currently in use by health bodies in the United Kingdom and available to others tackling the problem of evaluating digital health interventions pragmatically and responsively.