The Effect of Rater Severity On Person Ability Measure: A Rasch Model Analysis

Abstract

This paper presents a method for analyzing oral examinations with an extended, many-faceted Rasch model that calibrates medical specialty candidates, protocols, and raters. Significant variance was found among protocol difficulties and rater severities. When candidates' raw scores were compared with calibrated measures corrected for the bias caused by the particular protocols and raters encountered, variation between candidate scores and measures were observed. The data were found to fit the Rasch model well enough to be suitable for making measurement on oral examinations more objective as well as providing specific feedback to oral examination raters. In this example a medical oral examination was used; however, the techniques are applicable to any situation in which trained professionals rate candidate or patient performances. For occupational therapists, potential applications include evaluation of a student's fieldwork performance or observation of a patient's task performance.

Development of the Assisting Hand Assessment: A Rasch-built Measure intended for Children with Unilateral Upper Limb Impairments

“…It might also be of interest to consider whether the AHA could be extended to be suitable for older age groups by adding more challenging and age-appropriate tasks. A further area of future investigation is to determine whether the influence of differences of leniency of the raters affects the precision of the measures [42,43]. The instrument's responsiveness to changes in performance abilities seems promising, according to preliminary data from an ongoing intervention study, and is a very important issue for further investigation since we urgently need sensitive tools to evaluate the effectiveness of treatment in paediatric rehabilitation.…”

Intra- And Inter-Rater Reliability of the Assessment of Capacity for Myoelectric Control

“…In several studies (18,19) the variability between raters in terms of severity or leniency has been demonstrated. Raters seem to establish an individual profile of severity and usually tend to maintain this across clients and protocols (18).…”

“…Raters seem to establish an individual profile of severity and usually tend to maintain this across clients and protocols (18). The results from this study, with rater severity varying, in session 1 from (/0.51 (lenient) to 0.27 (severe) logits, and in session 2 from (/0.37 to 0.28 logits (Table II), and with a calibration difference of less than 0.16 logits, are in line with those findings.…”

The Presence and Impact of Local Item Dependence on Objective Structured Clinical Examinations Scores and the Potential Use of the Polytomous, Many-Facet Rasch Model

“…1 The many-facet Rasch model (MFRM) has been successfully used to adjust for rater stringency/leniency effects on candidate scores from orals and essays. [2][3][4][5] It has been suggested that the MFRM might be successfully applied to large-scale, high-stakes OSCEs. 6,7 It appears that the MFRM might be a useful tool for those engaged in OSCEs and assessments using rating systems where all candidates/students are not rated by the same rater(s).…”

“…Inconsistency of raters on oral examinations has been studied extensively. 2,10,[18][19][20][21][22] In 1993, Lunz and Stahl 10 reviewed an oral examination administered by the American Board of Urology. They found that examinees with identical raw scores often had differing ability measures after removing the effects because of rater stringency/leniency and case difficulty.…”