Standard setting anchor statements: a double cross-over trial of two different methods

Abstract

Context:We challenge the philosophical acceptability of the Angoff method, and propose an alternative method of standard setting based on how important it is for candidates to know the material each test item assesses, and not how difficult it is for a subgroup of candidates to answer each item. Methods:The practicalities of an alternative method of standard setting are evaluated here, for the first time, with direct comparison to an Angoff method. To negate bias due to any leading effects, a prospective cross-over design was adopted involving two groups of judges (n=7 and n=8), both of which set the standards for the same two 100 item multiple choice question tests, by the two different methods.Results: Overall, we found that the two methods took a similar amount of time to complete. The alternative method produced a higher cut-score (by 12-14%), and had a higher degree of variability between judges' cut-scores (by 5%). When using the alternative method, judges reported a small, but statistically significant, increase in their confidence to decide accurately the standard (by 3%). Conclusion:This is a new approach to standard setting where the quantitative differences are slight, but there are clear qualitative advantages associated with use of the alternative method.

Personalising knowledge assessments to remove compensation and thereby improve preparation for safe practice - developing content adaptive progress testing

“…If some people nd an item harder or easier than other people, then that does not make that item more or less important to be able to answer correctly. Using content enables a standard to be set with reference to importance without di culty (Burr et al, 2021). In addition, as all students in the same cohort have items selected randomly from the same pool, then this is fair as they all start with the same chance of getting any given item.…”