A Comparison of the Nedelsky and Angoff Cutting Score Procedures Using Generalizability Theory

Abstract

The American College Testing Program Nedelsky (1954) and Angoff (1971) have suggested procedures for establishing a cutting score based on raters' judgments about the likely performance of minimally competent examinees on each item in a test. In this paper generalizability theory is used to characterize and quantify expected variance in cutting scores resulting from each procedure. Experimental test data are used to illustrate this approach and to compare the two procedures. Consideration is also given to the impact of rater disagreement on some issues of measurement reliability or dependability. Results suggest that the differences between the Nedel sky and Angoff procedures may be of greater consequence than their apparent similarities. In particular, the restricted nature of the Nedelsky (inferred) probability scale may constitute a basis for seriously questioning the applicability of this procedure in certain contexts. Currently there is considerable debate concerning procedures for setting passing standards, or cutting scores, when scores on tests are used to make certain types of decisions (see, for example, National Council on Measurement in Education, 1978). Meskauskas (1976), Buck (1977), and Zieky and Livingston (1977), among others, have reviewed some current procedures for establishing cutting scores. For the most part these procedures can be grouped into two categories-procedures that use raters' subjective judgments and procedures that use examinee scores on the test itself and/or some criterion measure. The latter category of procedures is not discussed in this paper; rather, this paper examines two procedures suggested by Nedelsky (1954) and Angoff (1971) for establishing cutting scores based upon raters' judgments. Nedelsky and Angoff Procedures Both of these procedures require judgments by raters concerning the performance of hypothetical &dquo;minimally competent&dquo; examinees on each item of a test. Using Nedelsky's procedure, raters are asked to identify, for each item, those distractors that a minimally competent examinee would eliminate as incorrect. The reciprocal of the number of remaining alternatives (including the correct answer) serves as an estimate of the probability that a &dquo;minimally competent&dquo; examinee would get the item correct. In Angoff's prooedure, raters simply provide an estimate of the item probabilities with

An introduction to standard setting methods in dentistry

“…11,12 There is still no consensus as to the exact number of judges needed, and although previous studies have suggested a range of 5-20, most authors suggest that a group of 10 is an appropriate number. [16][17][18] Judges should also be knowledgeable of the curriculum that is being assessed, the abilities of the student cohort and should be ideally selected with a balanced mix of age, gender, educational experience and subject experience.…”

Estimating the Minimum Number of Judges Required for Test-centred Standard Setting on Written Assessments. Do Discussion and Iteration have an Influence?

“…This method of analysis has been described as 'essentially variance components analysis viewed from the perspective of generalizability' (Brennan and Lockwood, 1980) and variance components for before and after discussion Shaded values indicate the probability that the borderline candidate would correctly answer items in each category. These values were derived internally through a series of pilot trials of the Ebel method.…”

“…This provides a measure of the error associated with the pass mark for the paper, and indicates how another set of judges would set the pass mark for the paper (by generalising across the sample of judges used). The variance of the item effect is omitted from the error variance (Brennan and Lockwood, 1980;Verhoeven et al, 1999), and the root mean squared error (RMSE) estimated as:…”

Extending participation in standard setting: an online judging proposal

“…The Angoff method has been studied extensively (e.g. Brennan and Lockwood 1980;Livingston and Zieky 1989;Busch and Jaeger 1990;Chang 1999;Goodwin 1999;Giraud et al 2000;Hambleton 2001;MacCann 2008a). The Bookmark method is a newer method that is becoming very popular in the US and is attracting considerable research (e.g.…”