Quality control of an OSCE using generalizability theory and many-faceted Rasch measurement

Abstract

An Objective Structured Clinical Examination (OSCE) is an effective method for evaluating competencies. However, scores obtained from an OSCE are vulnerable to many potential measurement errors that cases, items, or standardized patients (SPs) can introduce. Monitoring these sources of errors is an important quality control mechanism to ensure valid interpretations of the scores. We describe how one can use generalizability theory (GT) and many-faceted Rasch measurement (MFRM) approaches in quality control monitoring of an OSCE. We examined the communication skills OSCE of 79 residents from one Midwestern university in the United States. Each resident performed six communication tasks with SPs, who rated the performance of each resident using 18 5-category rating scale items. We analyzed their ratings with generalizability and MFRM studies. The generalizability study revealed that the largest source of error variance besides the residual error variance was SPs/cases. The MFRM study identified specific SPs/cases and items that introduced measurement errors and suggested the nature of the errors. SPs/cases were significantly different in their levels of severity/difficulty. Two SPs gave inconsistent ratings, which suggested problems related to the ways they portrayed the case, their understanding of the rating scale, and/or the case content. SPs interpreted two of the items inconsistently, and the rating scales for two items did not function as 5-category scales. We concluded that generalizability and MFRM analyses provided useful complementary information for monitoring and improving the quality of an OSCE.

Psychometric evaluation of a knowledge based examination using Rasch analysis: An illustrative guide: AMEE Guide No. 72

“…Using the search terms: 'Rasch', 'Rasch analysis' and 'the multi-faceted Rasch model', in searching all medical education journal articles published between 1990 and January 2012 revealed only a few articles reporting on the application of the Rasch measurement model for analysing individual questions and individual OSCE stations (de Champlain et al 2003;Bhakta et al 2005;McManus et al 2006;Iramaneerat et al 2008;Houston 2009;Chang et al 2010;Yang et al 2011).…”

Back to the Future: Electronic Marking of Objective Structured Clinical Examinations and Admission Interviews Using an Online Management Information System in Schools of Health Sciences

“…However, current coefficients do not achieve the standards suggested in other research literature on the subject e.g. OSCEs [17,[19][20][21][22] The generalizability of results is only appropriate in OSCEs with a minimum of 15-18 stations [23][24][25].…”

A Comparison of Rubrics and Graded Category Rating Scales with Various Methods Regarding Raters’ Reliability

“…It is an extension of the basic Rasch model. It performs estimates by adding to the model any other sources of variability parameters affecting examinee performance besides rater severity and task difficulty (Iramaneerat et al, 2008;Linacre & Wright, 2004). The study included three sources of variability in MFRM, as in G theory.…”

“…MFRM is an extension of the basic Rasch model. The model estimates sources of variability of measurement by adding some parameters affecting examinee performance such as "raters" severity, task difficulty and any other sources of variability' (Iramaneerat, Yudkowsky, Myford, & Downing, 2008). By adding rater parameters into the measurement process, it becomes a useful instrument for estimating not only examinees' skills levels and the items' difficulty levels, but also the raters' severity levels as well (Linacre, Wright, & Lunz, 1990).…”

“…Besides, studies comparing methods used for determining inter-rater reliability based on different theories of measuring have an important place in recent research. These include studies comparing the methods based on the classical test theory, G theory, the many-facet Rasch measurement and the hierarchical rating model (Akın & Baştürk, 2010, 2012Engelhard, 1994;Engelhard & Myford, 2003;Güler & Gelbal, 2010b;Güler & Teker, 2015;Iramaneerart, Myford, Yudkowsky, & Lowenstein, 2009;Iramaneerat et al, 2008;Linacre et al, 1990;Lynch & McNamara, 1998;Macmillan, 2000;Nakamura, 2000;Stenlund, 2013;Sudweeks, Reeve, & Bradshaw, 2004). Further details are not provided in relation to the above mentioned studies since the present study aims at comparing rubrics and graded-category rating scales used in scoring rather than comparing the methods used to determine inter-rater reliability.…”