Reliability of a Minimal Competency Score for an Annual Skills Mastery Assessment

Abstract

Objective. To determine whether the modified Angoff process can be used to calculate a reliable minimal competency ("cut") score for the Annual Skills Mastery Assessment (ASMA). Methods. Three panels of pharmacy faculty members used a modified Angoff method to create a minimal competency score for 60 previously used test items. The panels did not know which items were included. Data were analyzed to determine differences between rating sessions, faculty type, item difficulty, and rater scoring bias. Results. The cut score generated was not significantly different by session or faculty type. The range of cut scores varied by less than 3% per examination. Faculty panelists correctly predicted student performance on items grouped as easy, medium, and hard. Conclusion. A properly constructed faculty panel can determine a reliable cut score and accurately rank relative test item difficulty using the modified Angoff process.

Determining defensible cut‐off scores for dental courses

“…Since each examination is 90‐120 items, the Angoff method of answering yes or no to each examination item (instead of the probability Angoff method that require raters to assign probability ratings ranging between 0.1 and 1.0) was selected. This is in alignment to what was done by Alston and Haltom where they conducted a study to determine whether the modified Angoff process can be used to calculate a reliable minimal competency “cut” score for the Annual Skills Mastery Assessment.…”

The methodology for the early identification of students at risk for failure in a professional degree program

“…The process for development of this exam has been previously explained in the literature 23 and the reliability of the mastery cut score has been analyzed. 24 The complete glossary of these dummy variables is described in Table 1, with the breakpoint values used to assign a value of 0 or 1 detailed. A three-phase approach was followed to analyze the data.…”

Awareness and performance of iranian nurses with regard to health economics: A cross-sectional study

“…The passing score was determined by using the Angoff method, which is one of the most common methods used for setting cutoff scores. [ 15 16 ] Each question is rated by a group of five subject matter experts, using criterion-referenced procedures. The expert estimates the percentage of minimally competent persons that will answer each question correctly, after which the judges’ totals are summed and averaged (8.75 + 7.25 + 8.5 + 7.75 + 6.75 divided by 5) to obtain a cutoff score of 7.8, or 8.…”