Development of the Engage with Impact Toolkit: A comprehensive resource to support the evaluation of patient, family and caregiver engagement in health systems

Abstract

Introduction Recent shifts in the patient, family and caregiver engagement field have focused greater attention on measurement and evaluation, including the impacts of engagement efforts. Current evaluation tools offer limited support to organizations seeking to reorient their efforts in this way. We addressed this gap through the development of an impact measurement framework and accompanying evaluation toolkit—the Engage with Impact Toolkit. Methods The measurement framework and toolkit were co‐designed with the Evaluating Patient Engagement Working Group, a multidisciplinary group of patient, family and caregiver partners, engagement specialists, researchers and government personnel. Project activities occurred over four phases: (1) project scoping and literature review; (2) modified concept mapping; (3) working group deliberations and (4) toolkit web design. Results The project scope was to develop a measurement framework and an evaluation toolkit for patient engagement in health systems that were practical, accessible, menu‐driven and aligned with current system priorities. Concept mapping yielded 237 impact statements that were sorted, discussed and combined into 81 unique items. A shorter list of 50 items (rated 8.0 or higher out of 10) was further consolidated to generate a final list of 35 items mapped across 8 conceptual domains of impact: (1) knowledge and skills; (2) confidence and trust; (3) equity and inclusivity; (4) priorities and decisions; (5) effectiveness and efficiency; (6) patient‐centredness; (7) culture change and (8) patient outcomes and experience. Working Group members rated the final list for importance (1–5) and identified a core set of 33 items (one for each of the 8 domains and 25 supplementary items). Two domains (priorities and decisions; and culture change) yielded the highest overall importance ratings (4.8). A web‐based toolkit ( www.evaluateengagement.ca ) hosts the measurement framework and related evaluation supports. Conclusion The Engage with Impact Toolkit builds on existing engagement evaluation tools but brings a more explicit focus to supporting organizations to assess the impacts of their engagement work. Patient Contribution Patient, family and caregiver partners led the early conceptualization of this work and were involved at all stages and in all aspects of the work. As end‐users of the toolkit, their perspectives, knowledge and opinions were critical.

PPIE in a technical research study: Using public involvement to refine the concept and understanding and move towards a multidimensional concept of disability

“…We offer concept mapping combined with co‐design, co‐production and public involvement as one such flexible combination of methods that researchers focusing on disability can employ to support public involvement. Other researchers have recently taken similar approaches combining concept mapping with collaborative methods in creating a patient engagement evaluation toolkit, 44 online group learning 45 and health promotion at gyms or health clubs. 46 …”

Co-design for stroke intervention development: Results of a scoping review

“…Finally, few studies evaluated their co-design methodology (e.g., impact on co-design participants, health programs, and organization) [ 161 ]. It may be valuable for researchers to continuously assess whether their co-design processes are working well and whether adjustments may be needed.…”

A holistic approach to integrating patient, family, and lived experience voices in the development of the BrainHealth Databank: a digital learning health system to enable artificial intelligence in the clinic

“…In retrospect, formal evaluations of the engagement were overlooked, which is a limitation of our work to date. Moving forward, implementing standardized evaluation tools and frameworks will help systematically evaluate the impact of each engagement and inform future engagements ( 38 , 39 ). Tools and frameworks such as the Public and Patient Engagement Evaluation Tool (PPEET) ( 40 ) and Guidance for Reporting Involvement of Patients and the Public (GRIPP2) ( 41 ) would allow for more consistent evaluations and reporting, thereby improving applicability across contexts.…”