CAEP 2019 Academic Symposium: Got competence? Best practices in trainee progress decisions

Cheung, Warren J.; Chan, Teresa M.; Hauer, Karen E.; Woods, Robert A.; McEwen, Jill; Martin, Lynsey J; Patocka, Catherine; Dong, Sandy L.; Bhimani, Munsif; McColl, Tamara

doi:10.1017/cem.2019.480

Cited by 5 publications

(6 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This study also identified variability in CCs’ abilities to maintain capacity among their members, which is aligned with the longstanding tension between resident education and service provision that has been reported in the literature 42,43 . This suggests a need to develop strategies for rewarding and managing faculty workload, including assigning merit points for individuals who serve on CCs, assigning a set number of reviewers per file, having members come to meetings having pre‐reviewed their files to increase efficiency, and where possible, rotating members on a regular basis 13,14,36 …”

Section: Discussionsupporting

confidence: 61%

“…Previous literature on group decision making has identified the importance of diverse committee membership in ensuring robust decision‐making processes, as this enables data to be considered from multiple lenses and helps reduce groupthink 7,11,13,36 . Nevertheless, the findings of the present study suggest that optimising CC membership is not always a simple task.…”

Section: Discussionmentioning

confidence: 71%

“…Although sampling was broad (ie across multiple programmes), it is recommended that future research take a multi‐site approach that compares CC implementation nationally and internationally. Moreover, since the purpose of this study was to document the challenges and adaptations of CCs, the findings represent considerations as opposed to consensus‐based recommendations 36 . Future studies might seek to evaluate which adaptations are effective in supporting CCs’ mandate and ensuring the integrity of their decision‐making processes.…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Competence committees: The steep climb from concept to implementation

et al. 2021

View full text Add to dashboard Cite

Introduction Competence committees (CCs) are groups of educators tasked with reviewing resident progress throughout their training, making decisions regarding the achievement of Entrustable Professional Activities and recommendations regarding promotion and remediation. CCs have been mandated as part of competency‐based medical education programmes worldwide; however, there has yet to be a thorough examination of the implementation challenges they face and how this impacts their functioning and decision‐making processes. This study examined CC implementation at a Canadian institution, documenting the shared and unique challenges that CCs faced and overcame over a 3‐year period. Methods This study consisted of three phases, which were conceptually and analytically linked using Moran‐Ellis and colleagues’ notion of ‘following a thread.’ Phase 1 examined the early perceptions and experiences of 30 key informants using a survey and semi‐structured interviews. Phase 2 provided insight into CCs’ operations through a survey sent to 35 CC chairs 1‐year post‐implementation. Phase 3 invited 20 CC members to participate in semi‐structured interviews to follow up on initial themes 2 years post‐implementation. Detailed observation notes from 16 CC meetings across nine disciplines were used to corroborate the findings from each phase. Results Response rates in each phase were 83% (n = 25), 43% (n = 15) and 60% (n = 12), respectively. Despite the high degree of support for CCs among faculty and resident members, several ongoing challenges were highlighted: adapting to programme size, optimising membership, engaging residents, maintaining capacity among members, sharing and aggregating data and developing a clear mandate. Discussion Findings of this study reinforce the importance of resident engagement and information sharing between disciplines. Challenges faced by CCs are discussed in relation to the existing literature to inform a better understanding of group decision‐making processes in medical education. Future research could compare implementation practices across sites and explore which adaptations lead to better or worse decision‐making outcomes.

show abstract

Section: Discussionsupporting

confidence: 61%

Section: Discussionmentioning

confidence: 71%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Competence committees: The steep climb from concept to implementation

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Even if we are able to significantly improve the usefulness of supervisor comments within a program, the trend of using increasing amounts of qualitative data brings on new challenges; narrative comments will take more time to analyze and integrate, and they will require skills that current faculty may not possess. 23,24 Given the importance of having useful supervisor narrative comments within CBME, 1,2 and that the QuAL demonstrates validity evidence for measuring utility, it is a tool that can serve multiple purposes. One aspect of potential program evaluation in CBME is to determine whether or not the narrative comments in a program or at an institution meet a certain standard over time.…”

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Validity evidence for the Quality of Assessment for Learning score: a quality metric for supervisor comments in Competency Based Medical Education

Woods

Singh²,

Thoma³

et al. 2022

Can. Med. Ed. J

Self Cite

View full text Add to dashboard Cite

Background: Competency based medical education (CBME) relies on supervisor narrative comments contained within entrustable professional activities (EPA) for programmatic assessment, but the quality of these supervisor comments is unassessed. There is validity evidence supporting the QuAL (Quality of Assessment for Learning) score for rating the usefulness of short narrative comments in direct observation. Objective: We sought to establish validity evidence for the QuAL score to rate the quality of supervisor narrative comments contained within an EPA by surveying the key end-users of EPA narrative comments: residents, academic advisors, and competence committee members. Methods: In 2020, the authors randomly selected 52 de-identified narrative comments from two emergency medicine EPA databases using purposeful sampling. Six collaborators (two residents, two academic advisors, and two competence committee members) were recruited from each of four EM Residency Programs (Saskatchewan, McMaster, Ottawa, and Calgary) to rate these comments with a utility score and the QuAL score. Correlation between utility and QuAL score were calculated using Pearson’s correlation coefficient. Sources of variance and reliability were calculated using a generalizability study. Results: All collaborators (n = 24) completed the full study. The QuAL score had a high positive correlation with the utility score amongst the residents (r = 0.80) and academic advisors (r = 0.75) and a moderately high correlation amongst competence committee members (r = 0.68). The generalizability study found that the major source of variance was the comment indicating the tool performs well across raters. Conclusion: The QuAL score may serve as an outcome measure for program evaluation of supervisors, and as a resource for faculty development.

show abstract