A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges

Abstract

This article addresses the problem of accounting overall multivariate chance-corrected interobserver agreement when targets have been rated by different sets of judges (not necessarily equal in number). The proposed approach builds on Janson and Olsson’s multivariate generalization of Cohen’s kappa but incorporates weighting for number of judges and applies an expression for expected disagreement suitable for the case with different judges. The authors suggest that the attractiveness of this approach to multivariate agreement measurement lies in the interpretability of the terms of expected and observed disagreement as average distances between observations, and that addressing agreement without regard to the covariance structure among variables has advantages in simplicity and interpretability. Correspondences to earlier approaches are noted, and application of the proposed measure is exemplified using hypothetical data sets.

Detecting Psychiatric Disorders in Preschoolers: Screening With the Strengths and Difficulties Questionnaire

“…To evaluate inter-rater reliability, 9% of the interview audio recordings were re-coded by blinded raters. Pairs of raters obtained the following multivariate inter-rater reliabilities; 35 Because SDQ taps into broad categories of problems instead of specific and rare disorders (e.g., tics), the PAPA diagnoses were joined together to create the following broad groupings of DSM-IV diagnoses; (1) a group of 'emotional disorders' consisting of major depression (MDD), dysthymia, depression not otherwise specified (NOS), separation anxiety disorder (SAD), generalized anxiety disorder (GAD), social phobia, specific phobia, agoraphobia, selective mutism and obsessive-compulsive disorder (OCD) (k=.82); (2) a group of 'behavioral disorders' consisting of attention-deficit/hyperactivity disorder (ADHD), oppositional defiant disorder (ODD) and conduct disorder (CD) (k=.84); (3) a group of 'emotional and behavioral disorders' comprising the disorders included in (1) and (2) (k=.86); and (4) a group of 'any disorder' consisting of disorders in (1) and (2) plus motor tics, vocal tics, trichotillomania, parasomnias and dyssomnias (k=.83). Given the high rate of encopresis in this age group, the diagnosis was excluded to prevent it from falsely influencing the scales estimated screening efficiency.…”

Sleep Disorders in Preschoolers

“…Nine percent (n 5 89) of the interview audio recordings were recoded by blinded raters. The multivariate interrater reliabilities 30 were as follows: primary insomnia (k 5 0.90), primary hypersomnia (k 5 0.78), nightmare disorder (k 5 1.0), and any sleep disorder (k 5 0.83). Importantly, because certain disorders are relatively uncommon and the reliability sample was randomly drawn, the interviews contained only a few cases of children with certain sleep disorders.…”

Variance Estimation of Nominal-Scale Inter-Rater Reliability with Random Selection of Raters

“…The use of the Euclidean distance allows for a natural generalization of kappa from nominal to ordinal and to interval data. Of particular interest is Janson and Olsson's (2004) proposal that releases the requirement of having the same group of raters rate each subject. Simon (2006) proposed an extension of the kappa statistic in the field of sequential observation data, where agreement is measured on the proper identification of the observation units to be rated as well as on their classification into categories.…”