Statistical Modeling of Expert Ratings on Medical Treatment Appropriateness

Abstract

This article uses latent structure analysis to model ordered category ratings by multiple experts on the appropriateness of indications for the medical procedure carotid endarterectomy. The statistical method used is a form of located latent class analysis, which combines elements of latent class and latent trait analysis. It assumes that treatment indications fall into distinct latent classes, with each latent class corresponding to a different level of appropriateness. The appropriateness rating of a treatment indication by a rater is assumed determined by the latent class membership of the indication, rating category thresholds of the rater, and random measurement error. The located latent class model has two alternative forms: a normal ogive form, which derives from the assumption of normally distributed measurement error, and a logistic approximation to the normal form. The approach has the following advantages for the analysis of ordered category ratings by multiple experts: ( 1 ) it assesses whether different raters base ratings on the same or different criteria; ( 2 ) it assesses rater bias-the tendency of some raters to make higher or lower ratings than others; ( 3 ) it characterizes rater differences in rating category definitions; ( 4 ) it provides theoretically based methods for combining the ratings of different raters; and ( 5 ) it provides a description of the distribution of the latent trait. The data examined are appropriateness ratings on 848 indications for carotid endarterectomy made by nine medical experts. The located latent class approach provides unique insights concerning the data. It identifies what appears to be a set of clear nonindications for carotid endarterectomy, but a corresponding set of clear indications is not evident. The results indicate that all raters measured a common latent trait of treatment appropriateness, but that some measured the trait better than others. Rater differences in overall bias and rating category definitions are evident. Two methods are used to combine raters' ratings. One uses ratings to calculate a continuous appropriateness score for each indication. The other uses ratings to assign indications to discrete outcome categories, each corresponding to a specific level of appropriateness. The located latent class approach for ordered category measures has possible applications besides the analysis of expert ratings, such as item analysis. Potential extensions of the model are discussed.KEY WORDS: Expert ratings; Health care; Latent class analysis; Latent trait analysis; Observer agreement; Ordered category data. I. INTRODUCTION1986), who had panels of national experts rate the appropriateness of several medical treatments. There is much recent concern about ensuring the approBecause of space limitations, only a brief synopsis of stapriateness of medical treatment. One proposed strategy for tistical methods for the analysis of multiple expert rating establishing treatment standards is to have expert panels re-data with ordered categories is provided. More...

A Latent Class Extension of Signal Detection Theory, with Applications

“…A widely analyzed data set (e.g., Dillan & Mulani, 1984;Uebersax, 1993c;Vermunt & Magidson, 2000) presented in Dillon and Mulani (1984;and in Vermunt & Magidson, 2000) provides an interesting example. The task of three observers was to determine whether each of 164 individuals were positive, neutral, or negative towards an object.…”

“…The latent class SDT model is closely related to located latent class models (Uebersax, 1993b(Uebersax, , 1993c and to a latent trait finite mixture model discussed by Uebersax (1993a). Using Uebersax's (1993c) For two classes, c = 1, 2, the above is equivalent to the two class SDT model with a logit link and the following relations between the SDT parameters and the located latent class parameters: c jk = jk , d j = 1, j = 1/(1.7␣ j ), B 1 = 0, and B 2 = 1. Uebersax (1993c) refers to ␣ j as a measurement error parameter; the above shows that it is equivalent to the inverse of the scale parameter j in the SDT model.…”

“…Using Uebersax's (1993c) For two classes, c = 1, 2, the above is equivalent to the two class SDT model with a logit link and the following relations between the SDT parameters and the located latent class parameters: c jk = jk , d j = 1, j = 1/(1.7␣ j ), B 1 = 0, and B 2 = 1. Uebersax (1993c) refers to ␣ j as a measurement error parameter; the above shows that it is equivalent to the inverse of the scale parameter j in the SDT model. Note that, because d j is restricted to be unity in the located latent class model, ␣ j plays the same role as the detection parameter in the latent class signal detection model; this simply reflects the arbitrary scale: in SDT, d j is the distance between the underlying distributions with the variance fixed, whereas in located latent class analysis, the distance B 2 -B 1 is fixed and ␣ j reflects the conditional precision (i.e., the variance of the underlying distributions).…”

The inter- and intra-observer reliability of a locomotion scoring scale for sheep

“…Latent trait models can handle multiple observers and use the theory 299 that observed ratings are a continuous latent trait or set of latent classes (Nelson and 300 Pepe, 2000). Unlike ICC, latent trait models do not assume equal spacing of 301 categories and can provide information on all components of observer agreement 302 (Uebersax, 1993 …”

Reliability of Essay Rating and Score Adjustment

“…Linacre (1988) and Lunz, Wright, and Linacre (1990) consider 3 reader severity as an additional facet of the Rasch model for polytomously scored items, and estimate severity of each reader. Rater reliability as a source of measurement error has been extensively studied in medical applications; see, Landis and Koch (1980), Tanner and Young (1985) and Uebersax (1993).…”