The Effect of Rating Unfamiliar Items on Angoff Passing Scores

Clauser, Jerome C.; Hambleton, Ronald K.; Baldwin, Peter

doi:10.1177/0013164416670983

Cited by 9 publications

(7 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, Clauser and Hambleton () reported on a study in which judges were asked to provide Angoff judgments for all items but were asked to mark items for exclusion if the content material was so unfamiliar that they believed they could not make a reasonable judgment. They reported that, after accounting for judge stringency and perceived item difficulty, items marked as unfamiliar received judgments slightly over .10 lower than expected on a zero‐to‐one probability scale.…”

mentioning

confidence: 99%

Effect of Content Knowledge on Angoff‐Style Standard Setting Judgments

Margolis

Mee

Clauser

et al. 2016

Educational Measurement

Self Cite

View full text Add to dashboard Cite

Evidence to support the credibility of standard setting procedures is a critical part of the validity argument for decisions made based on tests that are used for classification. One area in which there has been limited empirical study is the impact of standard setting judge selection on the resulting cut score. One important issue related to judge selection is whether the extent of judges' content knowledge impacts their perceptions of the probability that a minimally proficient examinee will answer the item correctly. The present article reports on two studies conducted in the context of Angoff-style standard setting for medical licensing examinations. In the first study, content experts answered and subsequently provided Angoff judgments for a set of test items. After accounting for perceived item difficulty and judge stringency, answering the item correctly accounted for a significant (and potentially important) impact on expert judgment. The second study examined whether providing the correct answer to the judges would result in a similar effect to that associated with knowing the correct answer. The results suggested that providing the correct answer did not impact judgments. These results have important implications for the validity of standard setting outcomes in general and on judge recruitment specifically.

show abstract

mentioning

confidence: 99%

Effect of Content Knowledge on Angoff‐Style Standard Setting Judgments

Margolis

Mee

Clauser

et al. 2016

Educational Measurement

Self Cite

View full text Add to dashboard Cite

show abstract

“…Standard setting judges in their study involved practicing physicians as well as non-physician PhDs (e.g., Professors of Microbiology). Their results concluded that when the non-expert judges were asked to determine the cutoff score of an unfamiliar item, they provide lower passing score than for a familiar item (15).…”

Section: Discussionmentioning

confidence: 99%

Evaluation of Two Applied Methods for Standard Setting in Undergraduate Medical Programme at the Faculty of Medicine, Suez Canal University

Kamal¹,

ElAraby²,

Kamel³

et al. 2018

EIMJ

View full text Add to dashboard Cite

Standard setting process is known to overcome the problem of the arbitrary pass score which is prone to underrate or overrate students. At Faculty of Medicine, Suez Canal University (FOM/ SCU), arbitrary 60% cutoff score is used in written exams. The objective of the study is to evaluate the efficacy of applying two standard setting methods in undergraduate medical programme of FOM/ SCU. Two standard setting methods (Angoff and Hofstee) were applied on the integrated multiple choice question (MCQ) exam of the third year FOM/SCU undergraduate students. Eleven judges from different specialties participated in this study. The cutoff score was determined by each method. Focus group discussion was conducted to get the judges' perception regarding standard setting process, importance, judges' selection and challenges. The determined cutoff scores using Angoff and Hofstee methods were 54.98% and 44% respectively. Thematic analysis of focus group discussion revealed judges' acceptance for standard setting application on all exams and they agreed on involving non-specialists in the judges' committee. There is a difference between the fixed cutoff score currently used at FOM/SCU and those determined using Angoff and Hofstee methods. Multidisciplinary judges are satisfied with this difference and they echoed the use of standard setting methods in improving assessment process.

show abstract

“…It is logical to assume that expert knowledge may influence perceptions of importance (Clauser et al, 2016), but Burr S, Martin T, Edwards J, Ferguson C, Gilbert K, Gray C, Hill A, Hosking J, Johnstone K, Kisielewska J, Milsom C, Moyes S, Rigby-Jones A, Robinson I, Toms N, Watson H, Zahra D MedEdPublish https://doi.org/10.15694/mep.2021.000032.1 there are several elements of importance that can be considered. Advocacy for the importance of each judge's own area of expertise in content knowledge may be moderated, to some extent by rating each question in more than one element of importance, for example, the links to other concepts, the frequency with which the concept is encountered, and the consequences of not understanding a concept (Neve et al, 2019).…”

Section: Acceptability Of Conceptualisation and The Need To Agree Weightingsmentioning

confidence: 99%

Standard setting anchor statements: a double cross-over trial of two different methods

Burr¹,

Martin²,

Edwards³

et al. 2021

MedEdPublish

View full text Add to dashboard Cite

Context:We challenge the philosophical acceptability of the Angoff method, and propose an alternative method of standard setting based on how important it is for candidates to know the material each test item assesses, and not how difficult it is for a subgroup of candidates to answer each item. Methods:The practicalities of an alternative method of standard setting are evaluated here, for the first time, with direct comparison to an Angoff method. To negate bias due to any leading effects, a prospective cross-over design was adopted involving two groups of judges (n=7 and n=8), both of which set the standards for the same two 100 item multiple choice question tests, by the two different methods.Results: Overall, we found that the two methods took a similar amount of time to complete. The alternative method produced a higher cut-score (by 12-14%), and had a higher degree of variability between judges' cut-scores (by 5%). When using the alternative method, judges reported a small, but statistically significant, increase in their confidence to decide accurately the standard (by 3%). Conclusion:This is a new approach to standard setting where the quantitative differences are slight, but there are clear qualitative advantages associated with use of the alternative method.

show abstract

The Effect of Rating Unfamiliar Items on Angoff Passing Scores

Cited by 9 publications

References 19 publications

Effect of Content Knowledge on Angoff‐Style Standard Setting Judgments

Effect of Content Knowledge on Angoff‐Style Standard Setting Judgments

Evaluation of Two Applied Methods for Standard Setting in Undergraduate Medical Programme at the Faculty of Medicine, Suez Canal University

Standard setting anchor statements: a double cross-over trial of two different methods

Contact Info

Product

Resources

About