Rating the Quality of Entrustable Professional Activities: Content Validation and Associations with the Clinical Context

Abstract

BACKGROUND: Entrustable professional activities (EPAs) have been developed to assess resident physicians with respect to Accreditation Council for Graduate Medical Education (ACGME) competencies and milestones. Although the feasibility of using EPAs has been reported, we are unaware of previous validation studies on EPAs and potential associations between EPA quality scores and characteristics of educational programs. OBJECTIVES: Our aim was to validate an instrument for assessing the quality of EPAs for assessment of internal medicine residents, and to examine associations between EPA quality scores and features of rotations. DESIGN: This was a prospective content validation study to design an instrument to measure the quality of EPAs that were written for assessing internal medicine residents. PARTICIPANTS: Residency leadership at Mayo Clinic, Rochester participated in this study. This included the Program Director, Associate program directors and individual rotation directors. INTERVENTIONS:The authors reviewed salient literature. Items were developed to reflect domains of EPAs useful for assessment. The instrument underwent further testing and refinement. Each participating rotation director created EPAs that they felt would be meaningful to assess learner performance in their area. These 229 EPAs were then assessed with the QUEPA instrument to rate the quality of each EPA. MAIN MEASURES: Performance characteristics of the QUEPA are reported. Quality ratings of EPAs were compared to the primary ACGME competency, inpatient versus outpatient setting and specialty type. KEY RESULTS: QUEPA tool scores demonstrated excellent reliability (ICC range 0.72 to 0.94). Higher ratings were given to inpatient versus outpatient (3.88, 3.66; p = 0.03) focused EPAs. Medical knowledge EPAs scored significantly lower than EPAs assessing other competencies (3.34, 4.00; p < 0.0001). CONCLUSIONS:The QUEPA tool is supported by good validity evidence and may help in rating the quality of EPAs developed by individual programs. Programs should take care when writing EPAs for the outpatient setting or to assess medical knowledge, as these tended to be rated lower.

A systematic review of 7 years of research on entrustable professional activities in graduate medical education, 2011–2018

“…However, this issue has been infrequently examined; most of the included studies did not assess the quality of their EPAs. Researchers have recognised the importance of producing high‐quality EPAs, with two tools for measuring EPA quality available . Linking scores on these tools to the methods used could provide more insight into the preferred methods for development, and in turn, the case for implementing EPAs will be strengthened if greater consideration is given to quality.…”

“…To answer this question, it is necessary to assess the quality of EPAs that are developed, and this would be an interesting and important area for future research, providing insight into what methods should be prioritised, and included in a potential standardised template for developing EPAs. Future researchers could use the EQual or the QUEPA tools to determine the quality of EPAs, in tandem with the QATSDD measure of methodological quality to explore the relationship between these variables. In turn, the validity of EPAs could be assessed by looking at the data from quality assessment tools such as those previously mentioned and linking the score on the tool with the number and type of methods used to develop the EPAs.…”

Specific entrustable professional activities for undergraduate medical internships: a method compatible with the academic curriculum

“…Even though many of them were implemented in the construction of our APROCs, we still need to test the benchmarks’ functionality as a supervision tool and as a resource for formative assessment of the UI. Post et al [ 8 ] presented an instrument to review EPAs that takes into consideration seven aspects that were useful when testing the viability of EPAs in pilot studies.…”

Faculty perspectives of an entrustable professional activity (EPA) framework in predoctoral dental education

“…The QUEPA includes 15 items that evaluate whether the EPA is focused, observable, realistic, generalizable, and able to incorporate multiple competencies; the EQual includes 14 items that evaluate whether the EPA is discrete, essential to the profession, and function as an educational tool. 25,26 Both rubrics have individuals rate the EPAs on a scale from 1 to 5; for the QUEPA this is from strongly disagree to strongly disagree, whereas the EQual provides descriptions for each of the scale points. The researchers decided to use a modified QUEPA instrument as it has been applied in multiple settings and is less cumbersome for an efficient evaluation.…”