Validity and reliability of an in-training evaluation report to measure the CanMEDS roles in emergency medicine residents

Abstract

Background: There is a question of whether a single assessment tool can assess the key competencies of residents as mandated by the Royal College of Physicians and Surgeons of Canada CanMEDS roles framework. Objective: The objective of the present study was to investigate the reliability and validity of an emergency medicine (EM) in-training evaluation report (ITER). Method: ITER data from 2009 to 2011 were combined for residents across the 5 years of the EM residency training program. An exploratory factor analysis with varimax rotation was used to explore the construct validity of the ITER. A total of 172 ITERs were completed on residents across their first to fifth year of training. Results: A combined, 24-item ITER yielded a five-factor solution measuring the CanMEDs role Medical Expert/ Scholar, Communicator/Collaborator, Professional, Health Advocate and Manager subscales. The factor solution accounted for 79% of the variance, and reliability coefficients (Cronbach alpha) ranged from a 5 0.90 to 0.95 for each subscale and a 5 0.97 overall. The combined, 24-item ITER used to assess residents' competencies in the EM residency program showed strong reliability and evidence of construct validity for assessment of the CanMEDS roles. Conclusion: Further research is needed to develop and test ITER items that will differentiate each CanMEDS role exclusively.

Validity of In-Training Evaluation Report (ITER) in Saudi Pediatric Residency Training. Single center experience

“…Our data concerning the validity of ITER were comparable with what has been published to date about this subject (Ginsburg, Eva & Regehr, 2013;Jackson, Kay & Frank, 2015;Kassam, Donnon & Rigby, 2014;Park, Riddle & Tekian, 2014), although Messick's validity evidence was not frequently used in most of these published studies. Our study found the ITER score to be valid and reliable, but it had limitations in detecting trainees with unprofessional conduct.…”

“…The validity of ITER or end-of-rotation evaluation was evaluated by several studies, and the majority of these studies found it to be a valid tool under certain conditions (Ginsburg, Eva & Regehr, 2013;Jackson, Kay & Frank, 2015;Kassam, Donnon & Rigby, 2014;Park, Riddle & Tekian, 2014). ITER also gives additional information about trainee progress over the years with regards to knowledge but not professionalism (Park, Riddle & Tekian, 2014) and despite its limitations in this regard, it is still a popular assessment tool (Pauls, 2012).…”

“…A retrospective study by Jackson et al (2015) looking at validity of ITER in 228 internal medicine residents revealed poor construct and predictive validity (Jackson, Kay & Frank, 2015). On the other hand, another study by Kassam et al (2014) found that ITER used in assessing medical emergency residents revealed strong reliability and evidence of construct validity (Kassam, Donnon & Rigby, 2014) while Ginsburg et al (2013) found that written comments by raters in ITER showed better correlation with trainee's performance than the scores did (Ginsburg, Eva & Regehr, 2013).Professionalism, which includes a list of attributes and behaviors, is a core competency of physicians and is expected by patients and medical societies (Mueller, 2015). Furthermore, since professionalism is associated with better clinical outcomes, medical trainees should be taught and assessed for professionalism (Mueller, 2015;Pauls, 2012).…”

Assessment of the Intrinsic CanMEDS Roles in Diagnostic Radiology Residents using an Objective Structured Clinical Assessment (OSCE)

“…The use of multiple assessment methods to gauge the Intrinsic CanMEDS competencies is favourable over the use of a single tool, such as a sole ITER (Kassam, et al, 2014). We must continue to explicitly teach and assess the Intrinsic CanMEDS Roles,…”

“…The current in-training evaluation reports (ITERs) used to make these assessments are subjective and can be based exclusively on the discretion of rotation supervisors. Other potential disadvantages of ITERs include small sample size of observations, variation in evaluator leniency, time between observation and assessment leading to recall bias, and assessment of clinical skills being influenced by the evaluator's own clinical skills (Kassam, et al, 2014;Kogan, et al, 2010).…”