Residency Program Director Evaluations Do Not Correlate With Performance on a Required 4th-Year Objective Structured Clinical Examination

Kahn, Marc J.; Merrill, William; Anderson, Delia; Szerlip, Harold M.

doi:10.1207/s15328015tlm1301_3

Cited by 33 publications

(31 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…13 Searches of the literature turned up other studies of relationships between OSCEs and scores on other evaluations (Table 4). 7,[9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24] In several examples the amounts of correlation appeared similar to the Pearson r values found in our study, while in several others the correlation appeared much lower, and only 1 example was found for which correlation appeared substantially higher than that of our study. 20 Some investigators' insights could be helpful to other investigators.…”

Section: Discussionsupporting

confidence: 80%

See 1 more Smart Citation

Correlation Between Student Performances on Course Level Integrated Clinical Skills Examinations and Objective Structured Clinical Examinations in a Chiropractic College Program

Russell

Hoiriis

Guagliardo

2012

Journal of Chiropractic Education

View full text Add to dashboard Cite

Purpose: This retrospective study measured correlation of student performance between 2 objective structured clinical examinations (OSCEs) and an introductory integrated clinical skills course that preceded the OSCEs. The hypothesis was that there would be a strong, positive correlation between the earlier level examinations and the upper level OSCE, high enough that earlier examinations could be viewed as predictors of upper level OSCE performance. Methods: Using student scores for 5 academic terms of upper level OSCEs for 2008-2009 (n = 208) and respective earlier scores, correlation coefficients were calculated for the upper level OSCE and Clinical Skills course, and upper and lower level OSCEs. Multiple linear regression analysis was used to evaluate how well the lower level OSCE and clinical skills scores, both as lone and combined independent variables, predicted the upper level OSCE scores. Results: There was at least a moderate correlation between both sets of scores: r = .51 (p < .001) between upper level OSCE and clinical skills course, r = .54 (p < .001) between the upper and lower level OSCEs. A combination of clinical skills and lower level OSCE scores suggested a moderate prediction of upper level OSCE scores (R 2 = .38.) Conclusions: Correlations were found to be of at least a moderate level. According to linear regression analysis, a combination of the earlier scores was moderately predictive for the upper level OSCE. More research could be done to determine additional components of student performance.

show abstract

Section: Discussionsupporting

confidence: 80%

“…Kahn et al, 15 2001, medical 4th year OSCE scores compared to evaluation of clinical skills by residency directors, class rankings, and medical licensing examination scores; no significant correlation of OSCE to composite of other assessments: r = .22, p = .15.…”

Section: Author Year Program Of Study Study Descriptions and Commentsmentioning

confidence: 98%

Correlation Between Student Performances on Course Level Integrated Clinical Skills Examinations and Objective Structured Clinical Examinations in a Chiropractic College Program

Russell

Hoiriis

Guagliardo

2012

Journal of Chiropractic Education

View full text Add to dashboard Cite

show abstract

“…Validated objective instruments are also needed to compare the efficacy and educational utility of proposed curricula, as evidenced by studies demonstrating poor correlation between self-assessment and assessment by evaluators with actual skill level and competence [20,21,22,23]. …”

Section: Background and Rationalementioning

confidence: 99%

Validation of Two Instruments to Assess Technical Bronchoscopic Skill Using Virtual Reality Simulation

2008

View full text Add to dashboard Cite

Background: In an era of increased emphasis on patient safety and competency-based education, demonstration of significant variability in the teaching of flexible bronchoscopy has led to initiatives for new standardized curricula and assessment tools. Objectives: To evaluate the efficacy of such curricula and to assess bronchoscopic skill, 2 measuring instruments have been developed: the Bronchoscopy Skills and Tasks Assessment Tool (BSTAT) and the Bronchoscopy Step-by-Step Evaluation Tool (BSET). We studied the validity and reliability of these 2 instruments. Methods: Two independent testers simultaneously scored 22 volunteer participants at 3 levels [novice (n = 7), fellow (n = 8) and attending (n = 7)] on a virtual reality bronchoscopy simulator using the 2 instruments; each participant was tested twice, in 2 separate sessions. Intertester and test-retest reliability were analyzed with intraclass correlations (ICC); ANOVA was used to assess concurrent validity based on the subjects’ expected skill level. Results: The ICCs between the testers were 0.98 for both the BSTAT and BSET. Comparison of the scores between the sessions showed high test-retest reliability by ICC (0.86 and 0.85 for BSTAT and BSET respectively), with a small yet statistically significant learning effect. The novice group’s scores were lower than the fellows’ and attendings’ (p < 0.001) for both the BSTAT and BSET; the fellows’ scores were consistently lower than the attendings’ on both tests, yet the differences were not statistically significant. Conclusion: This validation study of 2 objective tests of bronchoscopic skill demonstrated high reliability and concurrent validity. These instruments can now be used to evaluate the effectiveness of new competency-based bronchoscopy curricula.

show abstract

“…9 To obtain a high degree of reliability, at least 40 surveys are needed for each resident from each type of evaluator. [12][13][14] This is not a reasonable target, given the current construct of most residency programs. In 1 study, no significant correlation was found between the 360-degree evaluation during the first year of residency and the same subject's 4th-year objective measures including OSCE performance, class rank, and United States Medical Licensing Examination (USMLE) scores.…”

Section: Difficulties and Suggestions For Measuring Systemsbased Pracmentioning

confidence: 99%

“…In 1 study, no significant correlation was found between the 360-degree evaluation during the first year of residency and the same subject's 4th-year objective measures including OSCE performance, class rank, and United States Medical Licensing Examination (USMLE) scores. 12 The 360-degree evaluation may also be limited by language and literacy problems for patients and families. Another limiting factor for the use of 360-degree evaluations is that a fair amount of resources in staff and time must be allotted to collect, aggregate, and report survey responses.…”

Section: Difficulties and Suggestions For Measuring Systemsbased Pracmentioning

confidence: 99%

Development of the Objective Structured System-Interaction Examination

Hingle

Rosher²,

Robinson³

et al. 2009

Journal of Graduate Medical Education

View full text Add to dashboard Cite

Residency Program Director Evaluations Do Not Correlate With Performance on a Required 4th-Year Objective Structured Clinical Examination

Cited by 33 publications

References 11 publications

Correlation Between Student Performances on Course Level Integrated Clinical Skills Examinations and Objective Structured Clinical Examinations in a Chiropractic College Program

Correlation Between Student Performances on Course Level Integrated Clinical Skills Examinations and Objective Structured Clinical Examinations in a Chiropractic College Program

Validation of Two Instruments to Assess Technical Bronchoscopic Skill Using Virtual Reality Simulation

Development of the Objective Structured System-Interaction Examination

Contact Info

Product

Resources

About