Angoff and Nedelsky Standard Setting Procedures: Implications for the Validity of Proficiency Test Score Interpretation

Behuniak, Peter; Archambault, Francis X.; Gable, Robert K.

doi:10.1177/0013164482421031

Cited by 16 publications

(9 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The fact that the Angoff method has a tendency to set standards that are more difficult to meet has been observed by Paiva and Vu (1979) and Rock, Davis, and Werts (1980). Empirical findings confirming these tendencies can be found in Baron, Rindone, and Prowda (1981) ;Behuniak, Archambault, and Gable (1982); Brennan and Lockwood (1980); Cross et al (1984); Halpin, Sigmon, and Halpin (1983); Harasym (1981) ;Poggio, Glasnapp, and Eros (1981); Rock et al (1980);and Smith and Smith (1988). For a review of the empirical differences between Angoff and Nedelsky standard setting, see Chang (1999).…”

Section: Calculation Of Cutoff Scoresmentioning

confidence: 81%

Setting Standards and Detecting Intrajudge Inconsistency Using Interdependent Evaluation of Response Alternatives

Chang

Linden

Vos

2004

Educational and Psychological Measurement

View full text Add to dashboard Cite

This article introduces a new test-centered standard-setting method as well as a procedure to detect intrajudge inconsistency of the method. The standard-setting method that is based on interdependent evaluations of alternative responses has judges closely evaluate the process that examinees use to solve multiple-choice items. The new method is analyzed against existing methods, particularly the Nedelsky and Angoff methods. Empirical results from three different experiments confirm the hypothesis that standards set by the new method are higher than those of the Nedelsky but lower than those of the Angoff method. The procedure for detecting intrajudge inconsistency is based on residual diagnosis of the judgments, which makes it possible to identify the sources of inconsistencies in the items, response alternatives, and/or judges. An empirical application of the procedure in an experiment with the new standard-setting method suggests that the method is internally consistent and has also revealed an interesting difference between residuals for the correct and incorrect alternatives.A thorny and recurring issue in the standard-setting literature has been how to determine the validity of a standard set on an achievement test. Since the discussion on standard setting in the 1978 special issue of the Journal of Educational Measurement (Glass, 1978;Hambleton, 1978;Popham, 1978), the dominant view has been that a standard cannot be justified by any inde-

show abstract

Section: Calculation Of Cutoff Scoresmentioning

confidence: 81%

Setting Standards and Detecting Intrajudge Inconsistency Using Interdependent Evaluation of Response Alternatives

Chang

Linden

Vos

2004

Educational and Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…Once the MQ “profiles” have been drafted, SMEs are led through an Angoff 2 type process where a definition of the barely acceptable employee for a job is defined. SMEs then rate the remaining statements on two scales labeled “level” and “clarity.” An overview of Angoff approaches may be found in Angoff (1971) or Behuniak, Archambault, and Gable (1982).…”

Section: A Prior Approachmentioning

confidence: 99%

“…These percentages or proportions are summed over all items to generate the judge's estimate of a standard. The average of all judges' estimates should be used as the group's initial estimate of a standard.” (Behuniak, Archambault, & Gable, 1982).…”

mentioning

confidence: 99%

See 1 more Smart Citation

A Process for Content Validation of Education and Experienced‐based Minimum Qualifications: An Approach Resulting in Federal Court Approval

Buster¹,

Roth

Bobko³

2005

Personnel Psychology

View full text Add to dashboard Cite

The use of education and experience minimum qualifications (MQs) is nearly ubiquitous in employment settings, yet it appears to be rare that such MQs are validated by the end user (either via content validity or criterion‐related validation approaches). In this article, we present a method of content‐validating MQs that is related to a procedure noted by Levine, Maye, Ulm, and Gordon (1997), and we demonstrate the method's application for an upper‐level management position in a state agency. Our procedure is based on adherence to the Uniform Guidelines and sound professional practice. In addition, the procedure was heard in a federal court proceeding (deposition and expert testimony in relation to 3 jobs), was approved by that court, and we discuss the court's findings. In particular, the court found, “The MQ development process used by the State Personnel Department (SPD) is consistent with the requirements of the Guidelines and leads to content‐valid MQs.” Given data available to us for another job, we also show that the obtained MQs may result in less adverse impact than a previously determined, task‐based set of MQs.

show abstract

“…The most widely used standard setting method in medicine and health sciences is the the Angoff method, which has spawned several derivative methods (Hurtz and Auerbach 2003;Jalili et al 2011). Nonetheless, absolute/criterion-based methods such as Angoff (Behuniak et al 1982), Nelensky (1954, Ebel (1972) have significant shortcomings, namely:…”

Section: Introductionmentioning

confidence: 99%

The Objective Borderline method (OBM): a probability-based model for setting up an objective pass/fail cut-off score in medical programme assessments

Shulruf

Turner

Poole

et al. 2012

Adv in Health Sci Educ

View full text Add to dashboard Cite

The decision to pass or fail a medical student is a 'high stakes' one. The aim of this study is to introduce and demonstrate the feasibility and practicality of a new objective standard-setting method for determining the pass/fail cut-off score from borderline grades. Three methods for setting up pass/fail cut-off scores were compared: the Regression Method, the Borderline Group Method, and the new Objective Borderline Method (OBM). Using Year 5 students' OSCE results from one medical school we established the pass/fail cut-off scores by the abovementioned three methods. The comparison indicated that the pass/fail cut-off scores generated by the OBM were similar to those generated by the more established methods (0.840 ≤ r ≤ 0.998; p < .0001). Based on theoretical and empirical analysis, we suggest that the OBM has advantages over existing methods in that it combines objectivity, realism, robust empirical basis and, no less importantly, is simple to use.

show abstract

Angoff and Nedelsky Standard Setting Procedures: Implications for the Validity of Proficiency Test Score Interpretation

Cited by 16 publications

References 5 publications

Setting Standards and Detecting Intrajudge Inconsistency Using Interdependent Evaluation of Response Alternatives

Setting Standards and Detecting Intrajudge Inconsistency Using Interdependent Evaluation of Response Alternatives

A Process for Content Validation of Education and Experienced‐based Minimum Qualifications: An Approach Resulting in Federal Court Approval

The Objective Borderline method (OBM): a probability-based model for setting up an objective pass/fail cut-off score in medical programme assessments

Contact Info

Product

Resources

About