Estimating Reliability From a Single Administration of a Criterion‐referenced Test*

The IIP Examination: an Analysis of Group Performance 2009–2011

“…Classification consistency refers to how often a person taking an examination on multiple occasions would receive the same pass/fail decision. This report calculates decision consistency using Subkoviak's [8] method, which is relatively common when examining consistency in credentialing examinations.…”

Criterion Referenced Assessment: Establishing Content Validity of Complex Skills Related to Specific Tasks

“…Reliability estimates for internal consistency, test-retest correlation, and Subkoviak's (1976) CGA estimate for mastery tests are presented below. Descriptive statistics for the test takers are reported by group and subgroup.…”

“…A better way to estimate reliability for a standardized proficiency assessment is to report the probability of a mastery decision using the same cut score or criterion on a parallel test. Subkoviak's (1976) Coefficient of Agreement (CGA) estimate for a mastery decision reports the probability that the test takers would be assigned mastery on a parallel test to the first test, based on results from a single test administration.…”

Practical Impact of Predictor Reliability for Personnel Selection Decisions

“…Using analytic or simulation approaches, several prior studies have examined the effect of measurement reliability on selection accuracy, which refers to the extent that the same classification decision is made based on true score and observed score based on single tests (e.g., Bradlow & Wainer, 1998;Rudner, 2001;Subkoviak, 1976;Swaminathan, Hambleton, & Algina, 1974). …”

“…However, some of these methods are restrictive, such as applying only to dichotomously scored items, applying only to compensatory models, or lacking direct implications for prediction accuracy and subsequent utility estimation (e.g., Huynh, 1976;Livingston & Wingersky, 1979;Subkoviak, 1976). Livingston and Lewis (1995) introduced a classification accuracy estimation method that can be applied to more variety of conditions where many restrictive assumptions are not necessary.…”