The inter-rater and intra-rater reliability of a new Canadian oral examination format in anesthesia is fair to good

Kearney, Ramona; Puchalski, Stephen A.; Yang, Homer; Skakun, Ernest N.

doi:10.1007/bf03020520

Cited by 27 publications

(23 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Another possible way to try and improve the process would be for the same examiners to ask the same question to all candidates, that is, have the candidates move from one room to another during their exam. 3 This would mean that the examiners in the same room would ask only one question to multiple candidates. This would have the effect of standardizing the exam as well as assuring some degree of standardization in the scoring.…”

Section: Discussionmentioning

confidence: 99%

“…In a study of practice oral examinations using the RCPSC oral examination format, the mean IRR ranged from 0.477 to 0.791 (measured at two different study sites and at two different times). 3 However, the examiners were not truly grading in isolation, and not all examiners were experienced RCPSC examiners.…”

Section: Discussionmentioning

confidence: 99%

“…With regard to IRR, in two studies of mock anesthesia oral examinations, one resembling the ABA and one resembling the RCPSC examination, the IRR was reported as being acceptable. 2,3 With regard to examinee factors, a study of surgery residents demonstrated that a candidate's communication style may be a significant factor in his/her overall score. 5 It is therefore possible that some candidates may fail oral examinations due to problems with IRR and/or because they are unable to communicate effectively.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Poor inter-rater reliability on mock anesthesia oral examinations

Jacobsohn

Klock

Avidan

2006

Can J Anesth/J Can Anesth

View full text Add to dashboard Cite

Purpose: Inter-rater reliability (IRR) and communication skills are both important factors that have been shown to affect oral examination scores. This study was designed to test: 1) IRR of a group of American Board of Anesthesiology (ABA) and the Royal College of Physicians and Surgeons of Canada (RCPSC) graders who graded in true isolation; 2) the effect of teaching residents examination techniques. Methods:This was a randomized, pretest-posttest trial. Twenty-five residents did an initial oral examination (E1) resembling the ABA examination. They were then randomized into two groups, a routine education group, and an intervention group that was taught oral examination skills. Six weeks later they did another oral examination (E2). The videotaped examinations were subsequently scored by six experienced RCPSC and ABA graders. Results:There was very poor IRR on E1 (weighted Kappa = 0.166, intraclass correlation coefficient 0.243), which improved only slightly on E2 (weighted Kappa = 0.275, P = NS; intraclass correlation coefficient = 0.405, P < 0.01). Pass rate for graderpairs increased from E1 to E2 (15% vs 43%, P = 0.01). The improved pass rate on E2 occurred in both the routine education group and in the intervention group. There was no significant difference between RCPSC and ABA graders. Teaching examination skills per se did not improve performance, but this conclusion may be limited by the poor IRR. Practice orals do appear to improve performance on future examinations. Conclusions: Inter-rater reliability may be poor when graders score an oral examination in true isolation. Teaching candidates an oral examination communication and presentation technique did not appear to improve performance. Oral examination practice may be of value in training for future examinations. Objectif : Le coefficient d'objectivité (CO) et les habiletés de communication sont deux facteurs importants qui modifient les scores de l'examen oral. Notre étude voulait tester : 1) le CO d'un groupe d'examinateurs de l'American Board of Anesthesiology (ABA) et du Collège royal des médecins et chirurgiens du Canada (CRMCC) qui travaillent en isolement complet ; 2) l'effet de l'enseignement de techniques d'examen aux résidents. Méthode : L'essai randomisé comportait un prétest et un posttest. Vingt-cinq résidents ont passé un premier examen oral (E1) comme celui de l'ABA. Puis, ils ont été répartis en un groupe de formation courante, et un groupe d'intervention à qui on a présenté des techniques d'examen oral. Six semaines plus tard, ils ont passé un autre examen (E2). Les examens enregistrés sur bande vidéo ont été cotés ensuite par six examinateurs expérimentés du CRMCC et de l'ABA. Résultats

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Poor inter-rater reliability on mock anesthesia oral examinations

Jacobsohn

Klock

Avidan

2006

Can J Anesth/J Can Anesth

View full text Add to dashboard Cite

show abstract

“…Ces résultats se comparent favorablement à ceux de Kearney dans une étude utilisant un examen oral structuré pour des examens de pratique similaires à ceux qui sont couramment utilisés par le CRMCC. 4 Vingt examinateurs ont testé 26 résidents de deux programmes de résidence canadiens (sites A et B). Les questions standardisées ont été notées au moyen d'échelles d'évaluation globale comportant des critères de performance reconnus.…”

Section: Editorialsunclassified

“…This compares favourably with the results found by Kearney in a study using a structured oral examination format for practice examinations similar to that currently used by the RCPSC. 4 Twenty faculty examined 26 residents from two Canadian residency programs (sites A and B). Standardized questions were scored using global rating scales with anchored performance criteria.…”

mentioning

confidence: 99%

The oral examination process — gold standard or fool’s gold

Houston

Kearney

Savoldelli

2006

Can J Anesth/J Can Anesth

Self Cite

View full text Add to dashboard Cite

T HE oral examination process has been a traditional component of the certification process in anesthesia for over 70 years. 1 It is believed that the oral examination can be used to evaluate problem solving ability, communication and collaboration skills and expert content knowledge skills. In spite of its widespread acceptance, this examination has been criticized for its lack of reliability and validity and the high cost of its administration.Reliability is the measure of both the consistency and precision of a testing tool. The three main sources of variability (decreased reliability) in the oral examination process are: 1) examiner related variability; 2) examination related variability; and 3) candidate related variability. In their paper entitled "Poor interrater reliability on mock anesthesia oral examinations" in this edition of the Journal, Jacobsohn, Klock, Avidan and the Oral Examination Group present a study which demonstrates poor inter-rater reliability in a mock oral examination context with raters grading in true isolation. 2 Twenty-five residents were examined in a mock examination process resembling the American Board of Anesthesiology (ABA) format on two occasions six weeks apart (E1 and E2). The examinations were videotaped and scored by three experienced ABA examiners and three experienced Royal College of Physicians and Surgeons of Canada (RCPSC) examiners in isolation. The examiners were provided with a standardized scoring system and an educational package to aid with standard setting. The inter-rater reliability as determined by using intraclass correlation coefficients was poor: 0.243 (0.177-0.305) for E1 and 0.405 (0.331-0.470) for E2. For 48% of the candidates examined, the chance of passing or failing was examiner dependent.

show abstract