Faculty development for the evaluation system: a dual agenda

Oller, Kellee L.; T, Cuc; Ledford, Robert; O’Brien, Kevin

doi:10.2147/amep.s124004

Cited by 7 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, the Milestones 2.0 must be clearly articulated and reliably understood by faculty 8 . An evidence‐based methodology to foster the shared mental model, ensure high‐quality evaluations by faculty, and develop means to identify areas for improvement using learner feedback is suggested 9 . Three significant changes made to the original Milestones include updates to the content, revision of the rating system, and creation of a Supplemental Guide.…”

Section: Step 1: Training Facultymentioning

confidence: 99%

“…8 An evidence-based methodology to foster the shared mental model, ensure high-quality evaluations by faculty, and develop means to identify areas for improvement using learner feedback is suggested. 9 Three significant changes made to the original Milestones include updates to the content, revision of the rating system, and creation of a Supplemental Guide. Content updates focus on balancing the benefits of a shared mental model and developing common language around learning assessments across specialties while, most importantly, making the Milestones 2.0 specific to Geriatric Medicine trainees (Table 1).…”

Section: Step 1: Training Facultymentioning

confidence: 99%

See 1 more Smart Citation

Looking back, moving forward: A practical guide to implementing the updated ACGME geriatric medicine Milestones 2.0

Streiter

Foley

Bennett

et al. 2023

J American Geriatrics Society

View full text Add to dashboard Cite

The Accreditation Council for Graduate Medical Education (ACGME) developed the Milestones as a tool to aid trainee assessment based on the framework of the six core competencies of practice. Variability in the interpretation and application of the original Milestones prompted the ACGME to convene work groups within the different specialties and subspecialties to update the Milestones. The Geriatric Medicine work group was convened in 2019 with the goal of clarifying and simplifying the language of the Milestones, revising content to be specific to geriatrics, and developing supplemental resources to aid in implementation and use. We suggest using a practical, four-step process to implement the updated Milestones, called the Milestones 2.0, in fellowship programs by: (1) training faculty in the use of the Milestones 2.0, including an

show abstract

Section: Step 1: Training Facultymentioning

confidence: 99%

Section: Step 1: Training Facultymentioning

confidence: 99%

Looking back, moving forward: A practical guide to implementing the updated ACGME geriatric medicine Milestones 2.0

Streiter

Foley

Bennett

et al. 2023

J American Geriatrics Society

View full text Add to dashboard Cite

show abstract

“…Faculty development has been proposed as a way of improving rater bias, 39,40 although such interventions could still result in faculty scores that are more lenient 41 or more stringent. 42 It has been suggested that residency programs can identify a subset of faculty dedicated to learning and performance assessments, and make them responsible for the majority of resident assessments.…”

Section: Robustness and Dynamics Of Key Findingsmentioning

confidence: 99%

Examining the accuracy of residents’ self-assessments and faculty assessment behaviours in anesthesiology

et al. 2021

View full text Add to dashboard Cite

Background: Residents’ accurate self-assessment and clinical judgment are essential for optimizing their clinical skills development. Evidence from the medical literature suggests that residents generally do poorly at self-assessing their performance, often due to factors relating to learners’ personal backgrounds, cultures, the specific contexts of the learning environment and rater bias or inaccuracies. We evaluated the accuracy of anesthesiology residents’ self-assessed Global Entrustment scores and determined whether differences between faculty and resident scores varied by resident seniority, faculty leniency, and/or year of assessment. Methods: We employed variance components modeling techniques and analyzed 329 pairs of faculty and self-assessed entrustment scores among 43 faculty assessors and 15 residents. Using faculty scores as the gold standard, we compared faculty scores with residents’ scores (xi(faculty)–xi(resident)), and determined residents’ accuracy, including over- and under-confidence. Results: The results indicate that residents were respectively over- and under-confident in 10.9% and 54.4% of the assessments but more consistent in their individual self-assessments (rho = 0.70) than faculty assessors. Faculty scores were significantly higher (α = 0.396; z = 4.39; p < 0.001) than residents’ self-assessed scores. Being a lenient/dovish (β = 0.121, z = 3.16, p < 0.01) and a neutral (β = 0.137, z = 3.57, p < 0.001) faculty assessor predicted a higher likelihood of resident under-confidence. Senior residents were significantly less likely to be under-confident compared to junior residents (β = -0.182, z =-2.45, p < 0.05). The accuracy of self-assessments did not significantly vary during the two years of the study period. Conclusions: The majority of residents’ self-assessments were inaccurate. Our findings may help identify the sources of such inaccuracies.

show abstract

“…It is well known that assessment promotes improvement, and in medical education, evaluation is a fact of life [ 2 ]. Although many medical colleges have established a system for evaluating teaching quality, the reliability and validity of the final results are questioned [ 3 – 5 ]. What barriers prevent objective and accurate results from being obtained?…”

Section: Introductionmentioning

confidence: 99%

Barriers to obtaining reliable results from evaluations of teaching quality in undergraduate medical education

Zhang

et al. 2020

BMC Med Educ

View full text Add to dashboard Cite

Background Medical education is characterized by numerous features that are different from other higher education programmes, and evaluations of teaching quality are an integral part of medical education. Although scholars have made extensive efforts to enhance the quality of teaching, various issues unrelated to teaching that interfere with the accuracy of evaluation results remain. The purpose of this study is to identify the barriers that prevent objective and reliable results from being obtained during the evaluation process. Methods This study used mixed methods (3 data sources) to collect opinions from different stakeholders. Based on purposive sampling, 16 experts familiar with teaching management and 12 s- and third-year students were invited to participate in interviews and discussions, respectively. Additionally, based on systematic random sampling, 74 teachers were invited to complete a questionnaire survey. All qualitative data were imported into NVivo software and analysed using thematic analysis in chronological order and based on grounded theory. Statistical analyses of the questionnaire results were conducted using SPSS software. Results Sixty-nine valid questionnaires (93.24%) were recovered. A total of 29 open codes were extracted, and 14 axial codes were summarized and divided into four selective codes: evaluation preparation, the index system, the operation process, and the consequences of evaluation. The main barriers to obtaining reliable evaluation results included inadequate attention, unreasonable weighting, poor teaching facilities, an index without pertinence and appropriate descriptions, bad time-points, incomplete information on the system, lagged feedback, and disappointing result application. Almost all participants suggested lowering the weight of students as subjects, with a weight of 50–60% being appropriate. Students showed dissatisfaction with evaluation software, and the participants disagreed over the definition of good teaching and the management of student attendance. Conclusions This study reveals the difficulties and problems in current evaluations of teaching in medical education. Collecting data from multiple stakeholders helps in better understanding the evaluation process. Educators need to be aware of various issues that may affect the final results when designing the evaluation system and interpreting the results. More research on solutions to these problems and the development of a reasonable evaluation system is warranted.

show abstract

Faculty development for the evaluation system: a dual agenda

Cited by 7 publications

References 21 publications

Looking back, moving forward: A practical guide to implementing the updated ACGME geriatric medicine Milestones 2.0

Looking back, moving forward: A practical guide to implementing the updated ACGME geriatric medicine Milestones 2.0

Examining the accuracy of residents’ self-assessments and faculty assessment behaviours in anesthesiology

Barriers to obtaining reliable results from evaluations of teaching quality in undergraduate medical education

Contact Info

Product

Resources

About