The Risks of Thoroughness: Reliability and Validity of Global Ratings and Checklists in an OSCE

Cunnington, J. P. W.; Neville, Alan J.; Norman, Geoff

doi:10.1007/978-94-011-4886-3_41

Cited by 33 publications

(60 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This suggests that the inclusion of a global item on the SPIRS was justified, a view confirmed by other studies (Cunnington, Neville, andNorman 1997, Scheffer et al 2008). Crossley and Jolly (2012) noted that overall judgements may give a more reliable view.…”

Section: Discussionsupporting

confidence: 71%

“…one which will provide a summation of a student's overall skills. Studies investigating the use of global judgements (Boursicot and Roberts 2006, Cunnington, Neville, and Norman 1997, Scheffer et al 2008 have found that these may suit clinicians' understanding of clinical competency more than a list of items which attempt to separate discrete components of the competency. However, whilst these may be a time-efficient and appropriate method to assess communication and other professional skills, they may not provide the required level of detail to assist students in improving their performance (Scheffer et al 2008).…”

Section: Development Of Interview Assessment Toolmentioning

confidence: 99%

See 1 more Smart Citation

An Investigation of the Standardised Patient Interview Rating Scale (SPIRS) for the Assessment of Speech Pathology Students in a Simulation Clinic

2015

View full text Add to dashboard Cite

Standardised patients (SPs) are increasingly utilised in health sciences education to assist students in the development of clinical competencies, including interviewing skills. This study investigated the development and validation of a rating scale for formative assessment of speech pathology students in an interview with an SP. Participants in this study were 76 undergraduate speech pathology students and 10 clinical educators who participated in a simulated clinic module. As part of the module, pairs of students interviewed an SP portraying a parent of a child with speech delay. The Standardised Patient Interview Rating Scale (SPIRS) was developed to assess students' foundation clinical competencies of communication, interviewing and professional practice skills. Students' interviews were videotaped, rated individually on the SPIRS by the clinical educator, and later re-rated by an expert rater. Data were analysed to determine the content validity, internal consistency, and inter-rater reliability of the tool. In addition, descriptive statistics were used to report student performance levels. Results indicated that the SPIRS had good content validity and internal consistency but that there may be some redundancy in individual items. An acceptable level of inter-rater reliability was achieved. Students generally scored highly, with non-verbal communication being the easiest and professional practice the most difficult skill to demonstrate. The SPIRS was found to be an appropriate tool for formative assessment of students in this simulated clinic module. Recommendations for improving its reliability were made. Further research is required to investigate use of the SPIRS as an assessment tool in other contexts utilising standardised patients.

show abstract

Section: Discussionsupporting

confidence: 71%

Section: Development Of Interview Assessment Toolmentioning

confidence: 99%

An Investigation of the Standardised Patient Interview Rating Scale (SPIRS) for the Assessment of Speech Pathology Students in a Simulation Clinic

2015

View full text Add to dashboard Cite

show abstract

“…[14][15][16] In order to evaluate the validity of an OSCE as an effective battery to assess clinical competence, evidence concerning construct validity should be gathered as described widely in the medical literature. [17][18][19] Content evidence, clarity of instructions, station developer expertise, and adequacy of OSCE content in relation to curriculum objectives should be critically examined by an appropriate panel of experts.…”

Section: Discussionmentioning

confidence: 99%

Malaysian Pharmacy Students' Assessment of an Objective Structured Clinical Examination (OSCE)

Awaisu

Rahman

Mohamed

et al. 2010

American Journal of Pharmaceutical Education

View full text Add to dashboard Cite

Objective. To implement and determine the effectiveness of an objective structured clinical examination (OSCE) to assess fourth-year pharmacy students' skills in a clinical pharmacy course. Design. A 13-station OSCE was designed and implemented in the 2007-2008 academic year as part of the assessment methods for a clinical pharmacy course. The broad competencies tested in the OSCE included: patient counseling and communication, clinical pharmacokinetics (CPK), identification and resolution of drug-related problems (DRPs), and literature evaluation/drug information provision. Assessment. Immediately after all students completed the OSCE, a questionnaire containing items on the clarity of written instructions, difficulty of the tasks, perceived degree of learning gained and needed, and the suitability of the references or literature resources provided was administered. More than 70% of the students felt that a higher degree of learning was needed to accomplish the tasks at the 2 DRP stations and 2 CPK stations and the majority felt the written instructions provided at the phenytoin CPK station were difficult to understand. Although about 60% of the students rated OSCE as a difficult form of assessment, 75% said it should be used more and 81% perceived they learned a lot from it. Conclusion. Although most students felt that the OSCE accurately assessed their skills, a majority felt the tasks required in some stations required a higher degree of learning than they had achieved. This may indicate deficiencies in the students' learning abilities, the course curriculum, or the OSCE station design. Future efforts should include providing clearer instructions at OSCE stations and balancing the complexity of the competencies assessed.

show abstract

“…7 The agreement between global ratings is as high as the agreement between checklist items if the grader making the global assessment has adequate expertise. [11][12][13] The SP graders in these cases were senior medical students and laypersons without medical training. If the graders lack a sufficient level of expertise then the checklist items could be weighted to allow a more accurate global assessment.…”

Section: Discussionmentioning

confidence: 99%

Real-Time Standardized Participant Grading of an Objective Structured Clinical Examination

Stowe¹,

Gardner²

2005

Am J Pharm Educ

View full text Add to dashboard Cite

The objective structured clinical examination (OSCE) is a method of testing that uses standardized participants (SPs) to measure the clinical competence of students and focuses on observable behaviors to determine outcomes. Standardized participants act as patients or other health care providers in a standardized encounter to assess student performance. Faculty members typically serve as graders for pharmacy-based OSCEs. Standardized participant actors can accurately and reliably assess student performance. [1][2][3][4][5] The National Board of Medical Examiners now requires an OSCE as part of medical licensure, utilizing SPs as graders. 1,4,5 Although this method of assessment provides information difficult to obtain through traditional pencil-andpaper tests, it requires considerable financial resources and faculty time. A 3-case OSCE for approximately 80 students, with 10 minutes for preparation and 10 minutes for the encounter, takes approximately 7 hours to administer, with 9 examination rooms operating simultaneously. This involves a minimum of 9 real-time graders for the entire 7-hour testing period. By having SPs assess student performance, faculty graders would not be needed during the examination. This would make the examinations less costly and less time intensive for the faculty members. The objective of this study was to determine the reliability of using SPs as both actors and real-time graders for a therapeutics final examination using the OSCE format by comparing SP checklist scoring and examination standard setting to that of faculty members. METHODS Course DescriptionAll students enrolled in PhPr 4625, Therapeutics I, in 2003 participated in this evaluation. Therapeutics I is a 5-hour required course in the spring semester of the second professional year of the curriculum that consists of both didactic and problem-based learning. Therapeutics I consists of modules that address the pharmacotherapy of renal, cardiovascular, and gastrointestinal disorders. The final examination for the course was a 5-case, 200-point examination, with 3 cases presented in the traditional OSCE format and 2 cases presented in written form. The therapeutic skills checklist for each of the 5 cases were worth 30 points each. The communication component was worth 50 points, with 25 points split between the 3 OSCE cases (8.3 points per case) and 25 points split between the 2 written documentation cases (12.5 points per case). Objective. To determine the reliability of using standardized participant (SP) actors as real-time graders for an objective structured clinical examination (OSCE). Methods. All students enrolled in Therapeutics I participated in a 3-station OSCE. Faculty and SP actors graded student performance in real time using a dichotomous performance checklist. Standard setting for the OSCE was done using the borderline method. Inter-rater reliability between the faculty members and SP was determined. Results. Fifty-one students completed the morning examination and 30 completed the afternoon examination. The i...

show abstract

The Risks of Thoroughness: Reliability and Validity of Global Ratings and Checklists in an OSCE

Cited by 33 publications

References 6 publications

An Investigation of the Standardised Patient Interview Rating Scale (SPIRS) for the Assessment of Speech Pathology Students in a Simulation Clinic

An Investigation of the Standardised Patient Interview Rating Scale (SPIRS) for the Assessment of Speech Pathology Students in a Simulation Clinic

Malaysian Pharmacy Students' Assessment of an Objective Structured Clinical Examination (OSCE)

Real-Time Standardized Participant Grading of an Objective Structured Clinical Examination

Contact Info

Product

Resources

About