Intraclass Correlation Associated with Therapists: Estimates and Applications in Planning Psychotherapy Research

Abstract

It is essential that outcome research permit clear conclusions to be drawn about the efficacy of interventions. The common practice of nesting therapists within conditions can pose important methodological challenges that affect interpretation, particularly if the study is not powered to account for the nested design. An obstacle to the optimal design of these studies is lack of data about the intraclass correlation coefficient (ICC), which measures the statistical dependencies introduced by nesting. To begin the development of a public database of ICC estimates, the authors report ICCs for a variety outcomes reported in 20 psychotherapy outcome studies. The magnitude of the N = 495 ICC estimates varied widely across measures and studies. The authors provide recommendations regarding how to select and aggregate ICC estimates for power calculations and show how researchers can use ICC estimates to choose the number of patients and therapists that will optimize power. Attention to these recommendations will strengthen the validity of inferences drawn from psychotherapy studies that nest therapists within conditions.

Examining classroom influences on student perceptions of school climate: The role of classroom management and exclusionary discipline strategies

“…Although we adjusted estimates at the school-level, we did not have an adequate sample size to test school-level effects. Also, it is necessary to use the multilevel approach with nested data because correlation between participants is expected (Zucker, 1990) and even low levels of correlation, as measured by the intraclass correlation coefficient, can increase Type I error (Baldwin et al, 2011).…”

A Bayesian Approach to Multilevel Structural Equation Modeling With Continuous and Dichotomous Outcomes

“…In these cases, ICC estimates will suggest that the data are independent, and researchers who interpret these estimates might consequently-and misleadingly-believe that the hierarchical structure of the data can be ignored. Disregarding the multilevel nature of data with small but nonzero ICCs is problematic because it can cause an inflation of Type I errors (Baldwin et al, 2011;Murray & Hannan, 1990;Siddiqui, Hedeker, Flay, & Hu, 1996). In a Bayesian analysis, boundary estimates are not an issue because variance components can be given priors that shrink estimates away from zero (Chung, RabeHesketh, Dorie, Gelman, & Liu, 2013).…”

“…However, smaller ICCs are regularly seen in educational and behavioral research (e.g., Muthén, 1991), and ignoring the hierarchical structure of data with ICCs as small as .02 can lead to inflated Type I errors (Baldwin et al, 2011;Murray & Hannan, 1990;Siddiqui et al, 1996).…”

Estimating effect size when there is clustering in one treatment group

“…Because intraclass correlations are used in planning clusterrandomized experiments, there have been considerable efforts to develop empirical evidence about intraclass correlations. One form of evidence is based on intraclass correlation estimates from experiments that have already been conducted (see, e.g., Baldwin et al, 2011;Schnurr et al, 2007). Another form of evidence is based on secondary analyses of sample surveys that use cluster-sampling designs (see, e.g., Hedges & Hedberg, 2007).…”