Challenges in measuring ACGME competencies: considerations for milestones

Abstract

BackgroundMeasuring milestones, competencies, and sub-competencies as residents progress through a training program is an essential strategy in Accreditation Council for Graduate Medical Education (ACGME)’s attempts to ensure graduates meet expected professional standards. Previous studies have found, however, that physicians make global ratings often by using a single criterion.MethodsWe use advanced statistical analysis to extend these studies by examining the validity of ACGME International competency measures for an international setting, across emergency medicine (EM) and neurology, and across evaluators. Confirmatory factor analysis (CFA) models were fitted to both EM and neurology data. A single-factor CFA was hypothesized to fit each dataset. This model was modified based on model fit indices. Differences in how different EM physicians perceived the core competencies were tested using a series of measurement invariance tests.ResultsExtremely high alpha reliability coefficients, factor coefficients (> .93), and item correlations indicated multicollinearity, that is, most items being evaluated could essentially replace the underlying construct itself. This was true for both EM and neurology data, as well as all six EM faculty.ConclusionsEvaluation forms measuring the six core ACGME competencies did not possess adequate validity. Severe multicollinearity exists for the six competencies in this study. ACGME is introducing milestones with 24 sub-competencies. Attempting to measure these as discrete elements, without recognizing the inherent weaknesses in the tools used will likely serve to exacerbate an already flawed strategy. Physicians likely use their “gut feelings” to judge a resident’s overall performance. A better process could be conceived in which this subjectivity is acknowledged, contributing to more meaningful evaluation and feedback.

Improving plastic surgery resident education and quality of care with outcomes feedback using the surgery report card: An initial experience

“…While all residencies strive towards meeting and mastering these Core Competencies, measuring the achievement of these objectively can be difficult to ascertain. 3 Plastic surgery residency programs struggle to meet practice-based learning feedback requirement and experiences combining each of the competencies are seldom afforded. 4 As plastic surgeons, tracking and analyzing outcomes are essential to becoming better and improving the quality of care for patients.…”

Association between clinical competencies and mental health symptoms among frontline medical staff during the COVID-19 outbreak: A cross-sectional study

“…In recent years, clinical core competences, including clinical skills and patient care, mastery of medical knowledge, health promotion and disease prevention, information and management, professionalism, interpersonal communication, academic research, and teamwork have become key to de ning medical staff's ability worldwide [24][25][26]. High clinicians' competency scores have become measures of good clinical performance.…”

Current Milestones Assessment Practices, Needs, and Challenges of Program Directors: A Collective Case Study in a Pediatric Hospital Setting

“…However, this project created challenges for some training programs that needed more complex forms of assessment (e.g., unsupervised practices and/or procedural tasks that require multi-level assessment in various setting or skill-based assessment that requires measuring multiple integrated abilities such as knowledge, critical thinking, problem-solving, and decision-making skills in a given domain) and lacked proper measures reflecting the six competencies in a reliable or valid way [ 1 ]. While some of the competencies were easier to assess through valid and reliable testing (e.g., National Board Scores, Written Examinations, In-Training Examination Scores), others would be more challenging to assess as in the study conducted by Natesan et al [ 5 ] reporting that the competencies of professionalism, interpersonal, and communication skills can be hardly differentiated in the mind of evaluators. Furthermore, some other competencies were more subjective in nature (e.g., systems-based practice and practice-based learning and improvement) and posed additional problems from an assessment perspective (e.g., competencies that are hard to observe and infrequent opportunities to assess and/or milestones that are not covered in a specific curriculum) [ 6 ].…”