How Well Do Internal Medicine Faculty Members Evaluate the Clinical Skills of Residents?

Noel, Gordon L.; Herbers, Jerome E.; Caplow, Madlen P.; Cooper, Glinda S.; Pangaro, Louis N.; Harvey, Joan

doi:10.7326/0003-4819-117-9-757

Cited by 183 publications

(126 citation statements)

References 24 publications

Supporting

Mentioning

116

Contrasting

Unclassified

Order By: Relevance

“…Thus, the assessments made by the experts in our study were unreliable. Although our sample was small it concurs with data from other studies involving assessment of consulting skills (Kalet et al, 1992;Noel et al, 1992). It seems reasonable to conclude that unstructured global assessments are unreliable.…”

Section: Discussionsupporting

confidence: 70%

“…There is current interest in improving the communication skills of both under-and post-graduate doctors, since doctors who have not undergone specific training on how to give information are poor at doing so (Maguire et al, 1986a). However, inconsistencies may arise in teaching because, without structured guidance, experienced clinicians are unreliable when assessing clinical skills (Noel et al, 1992). Interrater reliability has been highlighted as particularly poor in the assessment of interviewing skills (Kalet et al, 1992), students have been scored on the basis of their liability rather than by specific behavioural skills they demonstrate.…”

mentioning

confidence: 99%

See 1 more Smart Citation

The development of a structured rating schedule (the BAS) to assess skills in breaking bad news

1999

View full text Add to dashboard Cite

Summary There has been considerable interest in how doctors break bad news, with calls from within the profession and from patients for doctors to improve their communication skills. In order to aid clinical training and assessment of the skills used in breaking bad news there is a need for a reliable, practical and valid, structured rating schedule. Such a rating schedule was compiled from agreed criteria in the literature. Video-taped recordings of simulated consultations breaking bad news were independently assessed by three raters using the schedule and compared to three experts who gave global ratings. The primary outcome measures were internal consistency of the schedule and level of agreement between raters. The internal consistency was high with a Cronbach's alpha of 0.93. Agreement between raters using the schedule was moderate to good. The majority of the variation in scores was due to the differences in skills demonstrated in the interviews. The agreement between raters not using the schedule was poor. The BAS provides a simple to use, reliable, and consistent rating schedule for assessing skills used in breaking bad news. It could be a valuable aid to teaching this difficult task.

show abstract

Section: Discussionsupporting

confidence: 70%

mentioning

confidence: 99%

The development of a structured rating schedule (the BAS) to assess skills in breaking bad news

1999

View full text Add to dashboard Cite

show abstract

“…15,16,20 Several factors may have contributed to inter-rater inconsistency or contributed to assessments that differed from the intended performance level(s). These factors include limited number of observations, halo effect, weak manipulations in the "videotaping" and scripts (although they reflected real-life scenarios), inadequate understanding of the performance criteria, and diversity in rater expertise.…”

Section: Discussionmentioning

confidence: 99%

“…For example, Noel et al measured the accuracy of student assessments made by medical faculty members (n5203), who viewed two clinical case simulations on videotape. 15 Using a validated scoring system, overall accuracy was calculated at 32% for the group using an open-ended evaluation form and 60% for the group using a structured evaluation form. More than half of the participants rated the students' performances in the two scenarios as satisfactory or superior.…”

Section: Introductionmentioning

confidence: 99%

Towards an Operational Definition of Clinical Competency in Pharmacy

Ried

Douglas

2015

American Journal of Pharmaceutical Education

View full text Add to dashboard Cite

Objective. To estimate the inter-rater reliability and accuracy of ratings of competence in student pharmacist/patient clinical interactions as depicted in videotaped simulations and to compare expert panelist and typical preceptor ratings of those interactions. Methods. This study used a multifactorial experimental design to estimate inter-rater reliability and accuracy of preceptors' assessment of student performance in clinical simulations. The study protocol used nine 5-10 minute video vignettes portraying different levels of competency in student performance in simulated clinical interactions. Intra-Class Correlation (ICC) was used to calculate inter-rater reliability and Fisher exact test was used to compare differences in distribution of scores between expert and nonexpert assessments. Results. Preceptors (n542) across 5 states assessed the simulated performances. Intra-Class Correlation estimates were higher for 3 nonrandomized video simulations compared to the 6 randomized simulations. Preceptors more readily identified high and low student performances compared to satisfactory performances. In nearly two-thirds of the rating opportunities, a higher proportion of expert panelists than preceptors rated the student performance correctly (18 of 27 scenarios). Conclusion. Valid and reliable assessments are critically important because they affect student grades and formative student feedback. Study results indicate the need for pharmacy preceptor training in performance assessment. The process demonstrated in this study can be used to establish minimum preceptor benchmarks for future national training programs.

show abstract

“…18 Global rating scales also discriminate poorly between learners and are poor at identifying specific educational deficiencies. 19 With the exception of internal medicine, the forms for global rating assessment are not standardized between training programs; therefore, we have limited data for systemwide analysis of this assessment modality. Other types of communication scoring forms have received more specific study.…”

Section: Summary Of Assessment Methodsmentioning

confidence: 99%

Assessment of Communication and Interpersonal Skills Competencies

Hobgood

Riviello

Jourıles

et al. 2002

Academic Emergency Medicine

View full text Add to dashboard Cite

Excellent communication and interpersonal (C-IP) skills are a universal requirement for a well-rounded emergency physician. This requirement for C-IP skill excellence is a direct outgrowth of the expectations of our patients and a prerequisite to working in the increasingly complex emergency department environment. Directed education and assessment of C-IP skills are critical components of all emergency medicine (EM) training programs and now are a requirement of the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project. In keeping with its mission to improve the quality of EM education and in response to the ACGME Outcome Project, the Council of Emergency Medicine Residency Directors (CORD-EM) hosted a consensus conference focusing on the application of the six core competencies to EM. The objective of this article is to report the results of this consensus conference as it relates to the C-IP competency. There were four primary goals: 1) define the C-IP skills competency for EM, 2) define the assessment methods currently used in other specialties, 3) identify the methods suggested by the ACGME for use in C-IP skills, and 4) analyze the applicability of these assessment techniques to EM. Ten specific communication competencies are defined for EM. Assessment techniques for evaluation of these C-IP competencies and a timeline for implementation are also defined. Standardized patients and direct observation were identified as the criterion standard assessment methods of C-IP skills; however, other methods for assessment are also discussed. Key words: communication; interpersonal skills; core competency; assessment. ACADEMIC EMERGENCY MEDICINE 2002; 9:1257-1269 Emergency department (ED) patient encounters are unique in medicine. They are often a single encounter for a new/unknown patient, and the duration and quality of the encounter are influenced by time pressures. As such, the emergency physician (EP) must quickly establish rapport and trust with the patient, gather information, assess the cause of the emergency, and design a treatment plan. The EP must also be able to effectively communicate findings and plans with the patient, the family, and other members of the health care team. Communication and interpersonal (C-IP) skills are a key component of a well-rounded EP. These skills

show abstract

How Well Do Internal Medicine Faculty Members Evaluate the Clinical Skills of Residents?

Abstract: A structured form improved the accuracy of observations of clinical skills, but faculty still disagreed in their assessments of clinical competence. If program directors are to certify residents' clinical competence, better and more standardized evaluation is needed.

Cited by 183 publications

References 24 publications

The development of a structured rating schedule (the BAS) to assess skills in breaking bad news

The development of a structured rating schedule (the BAS) to assess skills in breaking bad news

Towards an Operational Definition of Clinical Competency in Pharmacy

Assessment of Communication and Interpersonal Skills Competencies

Contact Info

Product

Resources

About