Pass‐fail Reliability for Tests With Cut Scores: A Simplified Method

Breyer, F. Jay; Lewis, Charles

doi:10.1002/j.2333-8504.1994.tb01612.x

Cited by 10 publications

(14 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some procedures have been developed mainly for dichotomous items (Hanson & Brennan, 1990; Huynh, 1976; Peng & Subkoviak, 1980; Subkoviak, 1976). Later, procedures were suggested for tests with polytomous items or mixtures of dichotomous and polytomous items, including procedures suggested by Breyer and Lewis (1994), Livingston and Lewis (1995), and Lee, Brennan, and Wan (2009).…”

mentioning

confidence: 99%

Classification Consistency and Accuracy for Complex Assessments Using Item Response Theory

Lee

2010

J Educational Measurement

141

View full text Add to dashboard Cite

In this article, procedures are described for estimating single-administration classification consistency and accuracy indices for complex assessments using item response theory (IRT). This IRT approach was applied to real test data comprising dichotomous and polytomous items. Several different IRT model combinations were considered. Comparisons were also made between the IRT approach and two non-IRT approaches including the Livingston-Lewis and compound multinomial procedures. Results for various IRT model combinations were not substantially different. The estimated classification consistency and accuracy indices for the non-IRT procedures were almost always lower than those for the IRT procedures.

show abstract

mentioning

confidence: 99%

Classification Consistency and Accuracy for Complex Assessments Using Item Response Theory

Lee

2010

J Educational Measurement

141

View full text Add to dashboard Cite

show abstract

“…The form level analyses results are based on the 3,750 first‐time candidates from accredited institutions in Canada who took the exam from 2007 to 2012. Table 1 shows descriptive statistics based on 300‐item forms, internal consistency, cut scores on the theta scale (i.e., passing score), pass rates, and three decision consistency statistics based on Hanson and Brennan, 16 Livingston and Lewis, 17 and Breyer and Lewis 18 . To calculate these decision consistency values, we used the University of Iowa's BB‐CLASS software (http://www.education.uiowa.edu/centers/casma/ computer‐programs) for the Hanson and Brennan and the Livingston and Lewis methods and Microsoft Excel for the Breyer and Lewis method.…”

Section: Resultsmentioning

confidence: 99%

Evaluating the Impact of Releasing an Item Pool on a Test's Empirical Characteristics

Buckendahl

Gerrow²

2016

Journal of Dental Education

View full text Add to dashboard Cite

Protecting the security of examination questions is an important task for high-stakes examining boards/agencies and university programs. To maintain the security of questions, examining boards and university programs use a combination of prevention, detection, and enforcement strategies. A common prevention strategy is to establish a number of controls on access to questions; however, restricting access can motivate examinees to try harder to reconstruct questions that may appear on future versions of the test. Moreover, access to study materials by some groups and not others can present a challenge to the fairness of examinations. The aim of this study was to evaluate the empirical stability of test characteristics. Specifically, the primary research objective was to investigate the empirical stability of the items and test forms of a written examination before and after a specific policy decision was implemented. As a response to both of these concerns, this article describes a study that evaluated how psychometric (i.e., statistical) properties of test forms and individual questions might be affected by publicly releasing a larger number of questions from an item question pool. A series of analyses were conducted, including item drift to evaluate stability of the characteristics. The results suggest that empirical characteristics of the test forms and individual questions have remained relatively stable since the release policy was implemented. Specifically, statistical properties of the test forms have continued to perform similarly to test forms that were constructed prior to the release. Although the results of this study were promising, the context of this specific testing program may have offered additional protections such as a limited number of administrations that others may not. Therefore, testing/examining agencies and university programs may want to consider this strategy with appropriate caution.

show abstract

“…Several approaches for estimating decision consistency based on a single administration of a test that consists of dichotomously scored items have been reported in the literature (Hanson & Brennan, 1990;Huynh, 1976;Subkoviak, 1976). Other approaches might be extended to allow estimating decision consistency for a test that consists of polytomously scored items (Breyer & Lewis, 1994;Livingston & Lewis, 1995). Brennan and Wan (2004, April) proposed an alternative bootstrap approach that can be applied to more complex assessment.…”

Section: Analysesmentioning

confidence: 99%

Rater Certification Tests: A Psychometric Approach

Attali

2019

Educational Measurement

View full text Add to dashboard Cite

Rater training is an important part of developing and conducting large‐scale constructed‐response assessments. As part of this process, candidate raters have to pass a certification test to confirm that they are able to score consistently and accurately before they begin scoring operationally. Moreover, many assessment programs require raters to pass a calibration test before every scoring shift. To support the high‐stakes decisions made on the basis of rater certification tests, a psychometric approach for their development, analysis, and use is proposed. The circumstances and uses of these tests suggest that they are expected to have relatively low reliability. This expectation is supported by empirical data. Implications for the development and use of these tests to ensure their quality are discussed.

show abstract

Pass‐fail Reliability for Tests With Cut Scores: A Simplified Method

Cited by 10 publications

References 8 publications

Classification Consistency and Accuracy for Complex Assessments Using Item Response Theory

Classification Consistency and Accuracy for Complex Assessments Using Item Response Theory

Evaluating the Impact of Releasing an Item Pool on a Test's Empirical Characteristics

Rater Certification Tests: A Psychometric Approach

Contact Info

Product

Resources

About