WBAs in UME—How Many Are Needed? A Reliability Analysis of 5 AAMC Core EPAs Implemented in the Internal Medicine Clerkship

Dunne, Dana; Gielissen, Katherine; Slade, Martin D.; Park, Yoon Soo; Green, Michael

doi:10.1007/s11606-021-07151-3

Cited by 7 publications

(13 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…17 At Yale, where a smaller cohort of trained assessors used the scales, generalizability studies using the Ottawa scale in the internal medicine clerkship found that variance attributed to the learner was substantially greater than that found in the VCU-based study. 18 In addition to these studies, the Core EPAs pilot studied the performance of both scales in a simulated setting using a series of video and paper-based vignettes. In this context, both scales performed relatively well in terms of inter-rater reliability and generalizability.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Revisiting the Core Entrustable Professional Activities for Entering Residency

Amiel¹,

Andriole²,

Biskobing³

et al. 2021

Academic Medicine

View full text Add to dashboard Cite

As the AAMC initiates a pilot for the Core Entrustable Professional Activities (EPAs) for Entering Residency, we are seeking baseline data from residency program directors about the readiness of graduates of LCME-accredited US medical schools to perform the 13 Core EPAs without direct supervision upon entry to residency. These EPAs are based on the work of a thirteen-member expert panel informed by the literature and by feedback from the academic medicine community. Your response will be helpful in establishing a baseline against which we can assess impact as some schools implement the Core EPAs for Entering Residency.Your participation in this project is voluntary. All responses are confidential. The data will be reported in aggregate by specialty type for research purposes. No individual respondent or individual program will be identified in any report of these data. This data collection activity has been reviewed according to AAMC policies and procedures and its Institutional Review Board and is considered to be minimal risk. The AAMC has taken extensive measures to ensure the security of the data and the confidentiality of the responses. Nevertheless, if individually identified data were made public, it could prove embarrassing. If you have any questions about your rights as a participant, contact the AAMC Office of Human Subjects Research Protection by email

show abstract

Section: Resultsmentioning

confidence: 99%

“…In this context, both scales performed relatively well in terms of inter-rater reliability and generalizability. 17,18…”

Section: Resultsmentioning

confidence: 99%

Revisiting the Core Entrustable Professional Activities for Entering Residency

Amiel¹,

Andriole²,

Biskobing³

et al. 2021

Academic Medicine

View full text Add to dashboard Cite

show abstract

“…The amount of unexplained variance can be explained by the nature of workplace-based assessments; EPAs performed during genuine clinical work cannot be standardized. This amount of unexplained variance is not dissimilar from other tools studied in medical education literature ( 47 , 48 ) and could be reduced in future analysis by recording and including more facets in the analysis.…”

Section: Discussionmentioning

confidence: 92%

A Tool to Assess Competence in Critical Care Ultrasound Based on Entrustable Professional Activities

Israel

Slade²,

Gielissen

et al. 2023

ATS Scholar

Self Cite

View full text Add to dashboard Cite

Background Existing assessment tools for competence in critical care ultrasound (CCUS) have limited scope and interrupt clinical workflow. The framework of entrustable professional activities (EPAs) is well suited to developing an assessment tool that is comprehensive and readily integrated into the intensive care unit (ICU) training environment. Objective This study sought to design an EPA-based tool to assess competence in CCUS for pulmonary and critical care fellows and to assess the validity and reliability of the tool. Methods Eight experts in CCUS met to define the core EPAs for CCUS. A nominal group technique was used to reach consensus. An assessment tool was created based on the EPAs with a modified Ottawa entrustability scale. Trained faculty evaluated pulmonary and critical care fellows using this tool in the ICU over a 6-month study period at a single institution. An assessment of validity of the EPA-based tool is made with four sources of validity evidence: content, response process, reliability, and relation to other variables. Reliability and response process data were generated using generalizability theory analysis to estimate sources of variance in entrustment scores. Analysis of response process validity and validity by relation to other variables was performed using regression models. Results Fifty-four assessments were recorded during the study period, conducted on 23 trainees by 13 faculty. Content validity of the tool was demonstrated using expert consensus and published guidelines from critical care societies to define the EPAs. Response process validity was demonstrated by the low variance in entrustment scores due to evaluators (0.086 or 6%) and high agreement between score and trainee self-assessment (regression coefficient, 0.82; P < 0.0001). Reliability was demonstrated by the high “true” variance in entrustment score attributable to the trainee: 0.674 or 45%. Validity by relation to other variables was demonstrated using regression analysis to show correlation between entrustment score and the number of times a fellow has performed an EPA (regression coefficient, 0.023; P < 0.0001). Conclusion An EPA-based assessment tool for competence in CCUS was created. We obtained sufficient validity evidence on three of the diagnostic EPAs. Procedural EPAs were infrequently assessed, limiting generalizability in this subgroup.

show abstract

“…The ideal number of observations to obtain for a given domain is an area of recent study. Dunne et al designed a student-driven mobile assessment platform using the modi ed OCAT on 5 core EPAs in an 8week internal medicine clerkship (19). They found that the estimated number of observations needed to reach a reliability index of 0.7 ranged between 9 and 11 for all EPAs except EPA5 which was sensitive to case complexity.…”

Section: Discussionmentioning

confidence: 99%

The “Growth Instant Feedback Tool (GIFTool)” Workplace-Based Assessment of Entrustable Professional Activity Performance: Preliminary Evidence for Validity

Osman

Bradshaw

Reinstein

et al. 2023

Preprint

View full text Add to dashboard Cite

Background Prior validity studies of workplace-based assessments for use with Entrustable Professional Activities have largely adopted a framework of supervision. We created a digital workplace-based assessment, the Growth-promoting Instant Feedback Tool (GIFTool), which utilizes a coaching framework and prompts supervisors with pre-determined feedback options so students can easily receive substantive real-time feedback. Our objectives were to assess the feasibility and validity evidence of the GIFTool. Methods With input from the literature and content experts, we created a digital application where supervisors assess the level of coaching required, one strength and one area for improvement in eight Entrustable Professional Activity-related domains. We explored data from one year through descriptive statistics and analyses focused on relations to other variables and fairness. Results Our development process contributed to the tool’s content validity. Students on average completed 28.8 assessments with 100% completion rate. We found the tool discriminated between learners within and between domains, with no differences by gender for all domains and for under-represented-in-medicine status for six domains. We found strengths and areas for improvement that were consistent with typical student trajectories. There were statistically significant correlations between the GIFTool and end-of-clerkship assessments (R values 0.27–0.51). Conclusions We found the GIFTool is a feasible tool with some promising evidence of validity.

show abstract

WBAs in UME—How Many Are Needed? A Reliability Analysis of 5 AAMC Core EPAs Implemented in the Internal Medicine Clerkship

Cited by 7 publications

References 29 publications

Revisiting the Core Entrustable Professional Activities for Entering Residency

Revisiting the Core Entrustable Professional Activities for Entering Residency

A Tool to Assess Competence in Critical Care Ultrasound Based on Entrustable Professional Activities

The “Growth Instant Feedback Tool (GIFTool)” Workplace-Based Assessment of Entrustable Professional Activity Performance: Preliminary Evidence for Validity

Contact Info

Product

Resources

About