Estimating effect size when there is clustering in one treatment group

Abstract

Some experimental designs involve clustering within only one treatment group. Such designs may involve group tutoring, therapy administered by multiple therapists, or interventions administered by clinics for the treatment group, whereas the control group receives no treatment. In such cases, the data analysis often proceeds as if there were no clustering within the treatment group. A consequence is that the actual significance level of the treatment effects is larger (i.e., actual p values are larger) than nominal. Additionally, biases will be introduced in estimates of the effect sizes and their variances, leading to inflated effects and underestimated variances when clustering in the treatment group is not taken into account. These consequences of clustering can seriously compromise the interpretation of study results. This article shows how information on the intraclass correlation can be used to obtain a correction for biases in the effect sizes and their variances, and also to obtain an adjustment to the significance test for the effects of clustering.

School-Based Group Mentoring and Academic Outcomes in Vulnerable High-School Students

“…Using a quasi-experimental design with propensity score weights, we were able to reduce treatment-selection bias that may result from nonrandomization of participants into the intervention and comparison groups (Austin & Stuart, 2015). Furthermore, the use of partially clustered analyses provided more accurate estimates of effect sizes compared to analyses that did not account for the clustering among program participants (Hedges & Citkowicz, 2015). In addition, this study included school records, which are more objective measures of academic outcomes than self-report surveys (Zimmerman, Caldwell, & Bernat, 2002).…”

“…We used Hedges and Citkowicz (2015) recommendations for estimating effect size that account for clustering of schools within program participants. The reported effect sizes below are corrected for partially nested models for program participants.…”

Individual and Community Factors Associated with Household Insecticide-Treated Bednet Usage in the Sunyani West District of Ghana Two Years after Mass Distribution

“…The study focused on individual-level analysis. ITN use and its associated factors were assessed for the individual respondent on the assumption that observed characteristics are independent of others [ 10 , 11 ]. The Sunyani West district was considered as a unit.…”

Defining Summer Gain Among Elementary Students With or at Risk for Reading Disabilities

“…Effect sizes were estimated as described by Hedges and Citkowicz (2015). Propensity score weighting is becoming a common methodology in education (e.g., D’Agostino et al, 2017; Morgan et al, 2010; Retelsdorf et al, 2012), and partially clustered randomized trails are becoming more common in education (Lohr et al, 2014).…”