Performance of intraclass correlation coefficient (ICC) as a reliability index under various distributions in scale reliability studies

Abstract

Many published scale validation studies determine inter‐rater reliability using the intra‐class correlation coefficient (ICC). However, the use of this statistic must consider its advantages, limitations, and applicability. This paper evaluates how interaction of subject distribution, sample size, and levels of rater disagreement affects ICC and provides an approach for obtaining relevant ICC estimates under suboptimal conditions. Simulation results suggest that for a fixed number of subjects, ICC from the convex distribution is smaller than ICC for the uniform distribution, which in turn is smaller than ICC for the concave distribution. The variance component estimates also show that the dissimilarity of ICC among distributions is attributed to the study design (ie, distribution of subjects) component of subject variability and not the scale quality component of rater error variability. The dependency of ICC on the distribution of subjects makes it difficult to compare results across reliability studies. Hence, it is proposed that reliability studies should be designed using a uniform distribution of subjects because of the standardization it provides for representing objective disagreement. In the absence of uniform distribution, a sampling method is proposed to reduce the non‐uniformity. In addition, as expected, high levels of disagreement result in low ICC, and when the type of distribution is fixed, any increase in the number of subjects beyond a moderately large specification such as n = 80 does not have a major impact on ICC.

A Reliability Check of Walkability Indices in Seoul, Korea

“…The agreement percentage and the Kappa statistic are typically used when addressing categorical variables. However, when continuous variables are considered, Pearson correlation coefficient (Pearson's R), Intraclass correlation coefficient (ICC), Brand-Altman plot with limits of agreement (LOA), and coefficient of variance can be used [57][58][59][60][61][62][63]. As the scale of measurement in this study is continuous, Pearson's R, Brand-Altman plot with LOA, and ICC were chosen as reliability measures.…”

“…This study uses the ICC to measure inter-method reliability between four indices of walkability. The ICC is used to determine the consistency of measurements (reliability): A higher ICC indicates greater consistency [62]. In measuring the degree of agreement between variables, Kappa statistics are used for categorical variables, whereas the ICC is used for numerical or quantitative variables [63].…”

Repeatability of radiomics and machine learning for DWI: Short‐term repeatability study of 112 patients with prostate cancer

“…Fedorov et al calculated ICC values for apparent diffusion coefficient maps calculated using mono‐exponential function for tumor region in 15 PCa patients scanned with 2 different scanners. Although differences in study subjects and applied MRI acquisition protocols of these studies make a direct comparison of ICC values difficult, these studies demonstrated that raw imaging signal and texture features can be repeatable . Recently, public access for prostate MRI of 15 subjects has been provided by Fedorov et al However, radiomic features for non‐Gaussian DWI models have not been evaluated in terms of short‐term repeatability using the same scanner parameters.…”

A quantitative grading system of vesicoureteral reflux by contrastenhanced voiding urosonography

“…The threshold level for statistical significance was p<0.05. The consistency between qualitative grading diagnoses and quantitative grading diagnoses of VUR was calculated using the weighted kappa coefficient and intraclass correlation coefficient (ICC) [23]. Kappa and ICC values were interpreted using the following criteria: ≤0.20, poor; 0.21-0.40, fair; 0.41-0.60, moderate; 0.61-0.80, good; 0.81-1.00, very good [24].…”