Real‐Time Feedback on Rater Drift in Constructed‐Response Items: An Example From the Golden State Examination

Hoskens, Machteld; Wilson, Mark

doi:10.1111/j.1745-3984.2001.tb01119.x

Cited by 44 publications

(46 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…First, only moderate stability of rater effects (r=0.6) was found across the two monitoring systems, somewhat worryingly suggesting that different impressions of rater performance could be given by the adoption of a particular system. Other studies too have shown instability in rater effects (Baird et al, 2013;Congdon & McQueen, 2000;Hoskens & Wilson, 2001;Harik et al, 2009;Lamprianou, 2006;Myford & Wolfe, 2009), which might be explained by small sample sizes in the monitoring checks.…”

Section: Discussionmentioning

confidence: 95%

“…In a study of quality checks on a mathematics high school examination, Wilson & Case (2000) found significant supervisor group training ('table leader') effects, but could not establish an association with the subsequent behaviour of raters within teams. Hoskens & Wilson (2001) also found supervisor effects in their analysis of rating quality checks for a high school Economics examination, with one team of raters being significantly more severe.…”

Section: Introductionmentioning

confidence: 94%

“…One of our research questions was whether the different quality assurance systems would produce the same rank-ordering of raters' accuracy, as researchers have recently been interested in the stability of measures of rater effects over time or over subjects (e.g., Congdon & McQueen, 2000;Hoskens & Wilson, 2001;Harik et al, 2009;Lamprianou, 2006;Myford & Wolfe, 2009). The data included 22 examinations, with different styles of scoring rubric.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Rater accuracy and training group effects in Expert- and Supervisor-based monitoring systems

Baird

Meadows

Leckie³

et al. 2015

Assessment in Education: Principles, Policy & Practice

View full text Add to dashboard Cite

General rightsThis document is made available in accordance with publisher policies. Please cite only the published version using the reference above. examinations, giving a total of 5,500 data points. Two rater-monitoring systems (Expert consensus scores and Supervisor judgment of correct scores) were utilized for all raters.Results showed significant group training (table leader) effects upon rater accuracy and these were greater in the expert consensus score monitoring system. When supervisor judgment methods of monitoring were used, differences between training teams (table leader effects) were under-estimated. Supervisor-based judgments of raters' accuracies were more widely dispersed than in the Expert consensus monitoring system. Supervisors not only influenced their teams' scoring accuracies, they over-estimated differences between raters' accuracies, compared with the Expert consensus system. Systems using supervisor judgments of correct scores and face-to-face rater training are, therefore, likely to under-estimate table leader effects and over-estimate rater effects.

show abstract

Section: Discussionmentioning

confidence: 95%

Section: Introductionmentioning

confidence: 94%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Rater accuracy and training group effects in Expert- and Supervisor-based monitoring systems

Baird

Meadows

Leckie³

et al. 2015

Assessment in Education: Principles, Policy & Practice

View full text Add to dashboard Cite

show abstract

“…For example, it would be of basic interest to determine if detection was invariant for a given set of observers over repeated sessions; similar values of d j across sessions would provide evidence that d j measures a stable characteristic. This is analogous to research in item response theory that has studied the invariance of item parameters across different groups of examinees (see Hambleton & Swaminathan, 1985;Lord, 1980) or over time (e.g., Hoskens & Wilson, 2001), and to research in confirmatory factor analysis that has studied the invariance of factor loadings across groups or time (e.g., Alwin & Jackson, 1981;Byrne, Shavelson, & Muthén, 1989). Note that, from the view via SDT, one would expect that discrimination might be invariant, but not the criteria (so there is only a partial invariance); also note that conclusions with respect to the criteria can differ, depending on which criteria measure is used, as discussed previously.…”

Section: Validationmentioning

confidence: 99%

A Latent Class Extension of Signal Detection Theory, with Applications

DeCarlo¹

2002

Multivariate Behavioral Research

View full text Add to dashboard Cite

A latent class extension of signal detection theory is presented and applications are illustrated. The approach is useful for situations where observers attempt to detect latent categorical events or where the goal of the analysis is to select or classify cases. Signal detection theory is shown to offer a simple summary of the observers' performance in terms of detection and response criteria. Implications of the view via signal detection for the training of raters are noted, as are approaches to validating the parameters and classifications. An extension of the signal detection model to more than two latent classes, with a simple restriction on the detection parameters, is introduced. Sample programs to fit the models using software for latent class analysis or software for second generation structural equation modeling are provided.In many situations in psychology, education, and medicine, observers attempt to detect or discriminate between two or more classes of events. When the events are observable, the methods of signal detection theory (SDT) can be used to obtain a measure of an observer's ability to detect or discriminate (see Macmillan & Creelman, 1991;Swets, 1996). Consider, for example, a simple test of recognition memory. A list of words can be presented during a study period and a combination of old words (from the list) and new words can be presented during a subsequent test, with the observers' task being to decide whether each word is old or new, or to rate their confidence that a word is old or new. From the perspective of SDT, the effect of an event (an old or new word) can be represented by a continuous latent variable, usually interpreted as an observer's perception of the event (e.g., the familiarity of the word), which is used together with a response criterion to arrive at a decision of old or new. The results can be summarized by a detection parameter, say d, which for the memory example is a measure of recognition memory strength, and by a parameter c that indicates the location of the response criterion; the location of the criterion is viewed as

show abstract

“…For example, the reliability of scores on the communication portion of the United States Medical Licensing Examination (USMLE Ò ) Clinical Skills Examination increased from 0.77 to 0.84 when an OLS model was used to adjust scores for differences in case-SP leniency (Harik et al 2009), which is roughly equivalent to increasing the length of the examination by 50%. A potential limitation of statistical adjustment is that a rater's leniency may fluctuate or drift over time (McKinley and Boulet 2004;Hoskens and Wilson 2001). However, prior research suggests that leniency indices for communication skills are fairly stable for 2-3 months (Harik et al 2009).…”

Section: Introductionmentioning

confidence: 97%

The impact of statistical adjustment on conditional standard errors of measurement in the assessment of physician communication skills

Raymond

Clauser

Furman

2010

Adv in Health Sci Educ

View full text Add to dashboard Cite

The use of standardized patients to assess communication skills is now an essential part of assessing a physician's readiness for practice. To improve the reliability of communication scores, it has become increasingly common in recent years to use statistical models to adjust ratings provided by standardized patients. This study employed ordinary least squares regression to adjust ratings, and then used generalizability theory to evaluate the impact of these adjustments on score reliability and the overall standard error of measurement. In addition, conditional standard errors of measurement were computed for both observed and adjusted scores to determine whether the improvements in measurement precision were uniform across the score distribution. Results indicated that measurement was generally less precise for communication ratings toward the lower end of the score distribution; and the improvement in measurement precision afforded by statistical modeling varied slightly across the score distribution such that the most improvement occurred in the upper-middle range of the score scale. Possible reasons for these patterns in measurement precision are discussed, as are the limitations of the statistical models used for adjusting performance ratings.

show abstract

Real‐Time Feedback on Rater Drift in Constructed‐Response Items: An Example From the Golden State Examination

Cited by 44 publications

References 21 publications

Rater accuracy and training group effects in Expert- and Supervisor-based monitoring systems

Rater accuracy and training group effects in Expert- and Supervisor-based monitoring systems

A Latent Class Extension of Signal Detection Theory, with Applications

The impact of statistical adjustment on conditional standard errors of measurement in the assessment of physician communication skills

Contact Info

Product

Resources

About