How to Set Cutoff Scores for Knowledge Tests Used in Promotion, Training, Certification, and Licensing

Biddle, Richard

doi:10.1177/009102609302200105

Cited by 29 publications

(29 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, Cascio et al noted a trend toward the use of elaborate methods for setting cutoffs, such as the use of expert judgments. Biddle (1993) described the legal issues involved in the use of such a judgmental method (the Angoff method) in setting cutoff scores for knowledge tests. He noted that the Supreme Court has upheld the modified Angoff method (to be discussed later) as a means for determining cutoff scores.…”

Section: Legal Issuesmentioning

confidence: 99%

“…The Angoff method (Angoff,197 1) has received by far the greatest attention in the personnel psychology literature, largely due to its proven legal defensibility (see Biddle, 1993). In the Angoff method judges are asked to estimate for each item the percentage of MCPs who could answer the item correctly or the likelihood that an MCP would answer the item correctly.…”

Section: Downloaded By [Queensland University Of Technology] At 20:21mentioning

confidence: 99%

See 1 more Smart Citation

Setting Cutoff Scores for Personnel Selection Tests: issues. Illustrations, and Recommendations

Truxillo¹,

Donahue²,

Sulzer³

1996

Human Performance

View full text Add to dashboard Cite

Cutoff scores are necessary in many public sector selection contexts. In this article, reasons for using cutoff scores, legal and professional guidelines, and methods for setting cutoff scores are examined. Recent literature on the Angoff method is reviewed, and 2 applications of this approach are explored in public sector contexts. In Study 1, 3 variations on the Angoff method applied to a multiple-choice test for fire chief were compared. In Study 2, standards were set for a physical ability test for firefighter using an Angoff-based method. Data and methods from both studies are discussed in terms of reliability, practicality, and the organizational issues involved in implementation. Finally, we review areas for future research that could improve the standard setting process, and we give recommendations for standard setters based on the current literature.Many public sector organizations are required to set cutoff scores for examinations. Although others do not require cutoff scores, cutoffs may be necessary for adminRequests for reprints should be sent to Donald M. Truxillo, TRUXILU). DONAHUE, SULZERistrative reasons. However, the appropriate way to establish cutoffs is far from clear based on research or on professional and legal standards.In this article we begin by discussing the reasons for setting cutoff scores. Next, after briefly reviewing the relevant legal and professional guidelines, we discuss alternative methods for setting cutoffs. We then review the recent literature on the Angoff method, a judgmend approach that has gaintd popularity because of its proven record of withstanding leg& scrutiny. We provide two examples of standard setting in the public sector: an application of the Angoff method to amultiple-choice promotional exam, and an illustration of another Angoff-bad method for a physical ability test for firefighter. We conclude with recommendations for research and practice.

show abstract

Section: Legal Issuesmentioning

confidence: 99%

Section: Downloaded By [Queensland University Of Technology] At 20:21mentioning

confidence: 99%

Setting Cutoff Scores for Personnel Selection Tests: issues. Illustrations, and Recommendations

Truxillo¹,

Donahue²,

Sulzer³

1996

Human Performance

View full text Add to dashboard Cite

show abstract

“…Biddle suggested the following recommendations to create a reliable examination cut score: 12 (1) use at least 7 to 10 subject matter experts as judges; (2) ask each judge to state the probability for each test item that the minimally acceptable person would answer the item correctly; (3) sum the judges' estimates for each test item, average the score per test item, and then sum the averages for the items on the test to create the cut score; (4) calculate the reliability and standard deviation for the test scores after the test is administered; and (5) consider the standard error of measurement before setting the final cut score.…”

Section: Introductionmentioning

confidence: 99%

Reliability of a Minimal Competency Score for an Annual Skills Mastery Assessment

Alston

Haltom

2013

American Journal of Pharmaceutical Education

View full text Add to dashboard Cite

Objective. To determine whether the modified Angoff process can be used to calculate a reliable minimal competency ("cut") score for the Annual Skills Mastery Assessment (ASMA). Methods. Three panels of pharmacy faculty members used a modified Angoff method to create a minimal competency score for 60 previously used test items. The panels did not know which items were included. Data were analyzed to determine differences between rating sessions, faculty type, item difficulty, and rater scoring bias. Results. The cut score generated was not significantly different by session or faculty type. The range of cut scores varied by less than 3% per examination. Faculty panelists correctly predicted student performance on items grouped as easy, medium, and hard. Conclusion. A properly constructed faculty panel can determine a reliable cut score and accurately rank relative test item difficulty using the modified Angoff process.

show abstract

“…A second potential derivation of Kane's (1987) index is to focus on the individual items' z-ratios within the brackets of Equation 2, treating them as onesample z-tests-or more accurately (because of relatively small recommended sample sizes for rater panels generally falling in the range of 7 to 15; Biddle, 1993;Hurtz & Hertz, 1999;Jaeger, 1991) one-sample t-tests which account for the number of raters in their degrees of freedom. With this approach one could simply count the proportion of items for which the rater panel's mean, or M iR , significantly differed from the value provided by the ICC, or P i (θ*).…”

Section: Indexes Based On the Standard Errors Of Ratingsmentioning

confidence: 99%

“…Others have addressed the conversion of ratings or cutoff scores to the θ scale via the item or test characteristic curves (Hurtz, Jones, & Jones, 2008;Kane, 1987;Plake & Kane, 1991;van der Linden, 1982). Given the widespread use of the classical Angoff-based methods and their general support in the face of professional and legal scrutiny (Berk, 1986;Biddle, 1993;Biddle, 2007;Cascio, Alexander, & Barrett, 1988;Kane, 1987;Maurer & Alexander, 1992), it is worthwhile to continue exploration of how well these classical methods perform in IRT-based examination programs. Van der Linden (1982) and Kane (1987) proposed methods for converting ratings that are given in the form of expected probabilities of answering items correctly (e.g., the Angoff (1971) and Nedelsky (1954) methods) to a θ-metric cutoff score, denoted here as θ*.…”

mentioning

confidence: 99%

Innovations in Measuring Rater Accuracy in Standard Setting: Assessing “Fit” to Item Characteristic Curves

Hurtz

Jones

2009

Applied Measurement in Education

View full text Add to dashboard Cite

Standard setting methods such as the Angoff method rely on judgments of item characteristics; item response theory empirically estimates item characteristics and displays them in item characteristic curves (ICCs). This study evaluated several indexes of rater fit to ICCs as a method for judging rater accuracy in their estimates of expected item performance for target groups of test-takers. Simulated data were used to compare adequately fitting ratings to poorly fitting ratings at various target competence levels in a simulated two stage standard setting study. The indexes were then applied to a set of real ratings on 66 items evaluated at 4 competence thresholds to demonstrate their relative usefulness for gaining insight into rater "fit." Based on analysis of both the simulated and real data, it is recommended that fit indexes based on the absolute deviations of ratings from the ICCs be used, and those based on the standard errors of ratings should be avoided. Suggestions are provided for using these indexes in future research and practice.

show abstract

How to Set Cutoff Scores for Knowledge Tests Used in Promotion, Training, Certification, and Licensing

Cited by 29 publications

References 5 publications

Setting Cutoff Scores for Personnel Selection Tests: issues. Illustrations, and Recommendations

Setting Cutoff Scores for Personnel Selection Tests: issues. Illustrations, and Recommendations

Reliability of a Minimal Competency Score for an Annual Skills Mastery Assessment

Innovations in Measuring Rater Accuracy in Standard Setting: Assessing “Fit” to Item Characteristic Curves

Contact Info

Product

Resources

About