A Nonparametric Approach to Estimate Classification Accuracy and Consistency

Lathrop, Quinn N.; Cheng, Ying

doi:10.1111/jedm.12048

Cited by 6 publications

(7 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although there were several conditions in characteristics of the example that could explain this discrepancy (e.g., poor model fit, carry-over effects in test–retest data, or prior exposure of respondents to CES-D items), a future direction would be to determine if this pattern appeared in other empirical applications of the CES-D. Also, both the Lee (2010) and Gonzalez and Pelham (2021) methods to estimate classification consistency assume that the item response model fits the data well, but as shown in our empirical example, this might not always be the case. Future studies will investigate how model misspecification impacts the estimation of classification consistency (MacCallum et al, 2001) and to evaluate whether nonparametric IRT approaches are more robust to model misspecification (Lathrop & Cheng, 2014). Similarly, the estimation of classification consistency indices assumes that we have true item parameters, but the item parameters are estimates that are paired with standard errors.…”

Section: Discussionmentioning

confidence: 99%

“…This has motivated the development of approaches to estimate classification consistency using data from a single administration of the measure. Some of these approaches use observed scores (e.g., Livingston & Lewis, 1995), parameters from item response models (Lee, 2010; Rudner, 2005), Bayesian approaches (Wheadon, 2014), and nonparametric methods (Lathrop & Cheng, 2014). In this article, we focus on the model-based approach by Lee (2010), popular in the area of educational measurement (Lathrop & Cheng, 2013), for three reasons.…”

Section: Classification Consistencymentioning

confidence: 99%

See 1 more Smart Citation

Estimating classification consistency of screening measures and quantifying the impact of measurement bias.

González¹,

Georgeson²,

Pelham³

et al. 2021

Psychological Assessment

View full text Add to dashboard Cite

Screening measures are used in psychology and medicine to identify respondents who are high or low on a construct. Based on the screening, the evaluator assigns respondents to classes corresponding to different courses of action: Make a diagnosis versus reject a diagnosis; provide services versus withhold services; or conduct further assessment versus conclude the assessment process. When measures are used to classify individuals, it is important that the decisions be consistent and equitable across groups. Ideally, if respondents completed the screening measure repeatedly in quick succession, they would be consistently assigned into the same class each time. In addition, the consistency of the classification should be unrelated to the respondents' background characteristics, such as sex, race, or ethnicity (i.e., the measure is free of measurement bias). Reporting estimates of classification consistency is a common practice in educational testing, but there has been limited application of these estimates to screening in psychology and medicine. In this article, we present two procedures based on item response theory that are used (a) to estimate the classification consistency of a screening measure and (b) to evaluate how classification consistency is impacted by measurement bias across respondent groups. We provide R functions to conduct the procedures, illustrate the procedures with real data, and use Monte Carlo simulations to guide their appropriate use. Finally, we discuss how estimates of classification consistency can help assessment specialists make more informed decisions on the use of a screening measure with protected groups (e.g., groups defined by gender, race, or ethnicity). Public Significance StatementIdeally, measures used to classify individuals in psychology and medicine (e.g., as depressed vs. not depressed) produce consistent decisions that do not depend on extraneous features of the individuals (e.g., sex, race, or ethnicity). We provide methods to estimate how consistent the decisions based on a measure are and study whether the consistency varies across groups (e.g., sex, race, or ethnicity).

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Classification Consistencymentioning

confidence: 99%

Estimating classification consistency of screening measures and quantifying the impact of measurement bias.

González¹,

Georgeson²,

Pelham³

et al. 2021

Psychological Assessment

View full text Add to dashboard Cite

show abstract

“…Importantly, procedures used to estimate the precision of a pass-fail decision are usually referred to as methods for estimating a given test's classification accuracy. This issue has been reported on extensively in the broader literature on educational measurement and psychometrics (Huynh 1990;Kane 1996;Lathrop and Cheng 2014;Lee 2010;Lewis and Sheehan 1990;Subkoviak 1976;Rudner 2005;Webb et al 2006;Wyse and Hao 2012). These works, and especially the approach described by Rudner (2005), form the psychometric background for the following illustrations.…”

Section: Introductionmentioning

confidence: 95%

How sure can we be that a student really failed? On the measurement precision of individual pass-fail decisions from the perspective of Item Response Theory

Schauber

Hecht

2020

Medical Teacher

View full text Add to dashboard Cite

“…When comparing the quality of different measurement scales of internet addiction, Zhang and Xin ( 2013 ) used Lee's IRT procedure to obtain the CC indices. In addition, Lathrop and Cheng ( 2014 ) recently developed a new approach to estimate CC index non-parametrically by replacing the role of the parametric IRT model with a modified version of Ramsay's kernel-smoothed item response functions. However, their results showed that the non-parametric CC index performs similarly to Lee's procedure especially when the ability distributions are non-normal.…”

Section: Introductionmentioning

confidence: 99%

Using Procedure Based on Item Response Theory to Evaluate Classification Consistency Indices in the Practice of Large-Scale Assessment

Zhang

Chen

et al. 2017

Front. Psychol.

View full text Add to dashboard Cite

In spite of the growing interest in the methods of evaluating the classification consistency (CC) indices, only few researches are available in the field of applying these methods in the practice of large-scale educational assessment. In addition, only few studies considered the influence of practical factors, for example, the examinee ability distribution, the cut score location and the score scale, on the performance of CC indices. Using the newly developed Lee's procedure based on the item response theory (IRT), the main purpose of this study is to investigate the performance of CC indices when practical factors are taken into consideration. A simulation study and an empirical study were conducted under comprehensive conditions. Results suggested that with negatively skewed distribution, the CC indices were larger than with other distributions. Interactions occurred among ability distribution, cut score location, and score scale. Consequently, Lee's IRT procedure is reliable to be used in the field of large-scale educational assessment, and when reporting the indices, it should be treated with caution as testing conditions may vary a lot.

show abstract

A Nonparametric Approach to Estimate Classification Accuracy and Consistency

Cited by 6 publications

References 37 publications

Estimating classification consistency of screening measures and quantifying the impact of measurement bias.

Estimating classification consistency of screening measures and quantifying the impact of measurement bias.

How sure can we be that a student really failed? On the measurement precision of individual pass-fail decisions from the perspective of Item Response Theory

Using Procedure Based on Item Response Theory to Evaluate Classification Consistency Indices in the Practice of Large-Scale Assessment

Contact Info

Product

Resources

About