Evaluation Utility Metrics (EUMs) in Reflective Practice

Abstract

The article proposes three evaluation utility metrics to assist evaluators in evaluating the quality of their evaluation. After an overview of reflective practice in evaluation, the different ways in which evaluators can hold themselves accountableare discussed. It is argued that reflective practice requires evaluators to go beyondevaluation quality (i.e., technical quality and methodological rigor) when assessingevaluation practice to include an evaluation of evaluation utility (i.e., specific actions taken in response to evaluation recommendations). Three Evaluation Utility Metrics (EUMs) are proposed to evaluate utility: whether recommendations are considered(EUMc), adopted (EUMa), and (if adopted) the level of influence of recommendations (EUMli). The authors then reflect on their experience in using the EUMs, noting the importance of managing expectations through negotiation to ensure thatEUM data are collected and the need to consider contextual nuances (e.g., adoptionand influence of recommendations are influenced by multiple factors beyond thecontrol of the evaluators). Recommendations for increasing EUM rates by payingattention to the frequency and timing of recommendations are also shared. Resultsof implementing these EUMs in a real-world evaluation provide evidence of theirpotential value: practice tips led to an EUMc of 100% and and EUMa of over 80%.Methods for considering and applying all three EUMs together to facilitate practiceimprovement are also discussed.

Evaluating governance in a clinical and translational research organization

“…” Reflecting on the level of influence of a recommendation is germane to all IDeA CTRs. Generalization is challenging as findings depend on the system, culture, and context wherein governance recommendations were generated [16].…”

“…Our approach to how the GP IDeA-CTR responded to governance recommendations distinguished between recommendations to improve administrative and governance functions; CTR processes, capabilities, and capacity; and governance evaluation. Meta-evaluation in another CTR used “evaluation utility metrics” (EUMs) to classify whether recommendations were fully or partially adopted, and if they led to actionable changes in program processes or products [ 16 ].” Reflecting on the level of influence of a recommendation is germane to all IDeA CTRs. Generalization is challenging as findings depend on the system, culture, and context wherein governance recommendations were generated [ 16 ].…”

“…Meta-evaluation in another CTR used “evaluation utility metrics” (EUMs) to classify whether recommendations were fully or partially adopted, and if they led to actionable changes in program processes or products [ 16 ].” Reflecting on the level of influence of a recommendation is germane to all IDeA CTRs. Generalization is challenging as findings depend on the system, culture, and context wherein governance recommendations were generated [ 16 ].…”