Do therapist effects really impact estimates of within-patient mechanisms of change? A Monte Carlo simulation study

Abstract

Existing evidence highlights the importance of modeling differential therapist effectiveness when studying psychotherapy outcome. However, no study to date examined whether this assertion applies to the study of within-patient effects in mechanisms of change. The study investigated whether therapist effects should be modeled when studying mechanisms of change on a within-patient level. Methods: We conducted a Monte Carlo simulation study, varying patient-and therapist level sample sizes, degree of therapist-level nesting (intra-class correlation), balanced vs. unbalanced assignment of patients to therapists, and fixed vs random within-patient coefficients. We estimated all models using longitudinal multilevel and structural equation models that ignored (2-level model) or modeled therapist effects (3-level model). Results: Across all conditions, 2-level models performed equally or were superior to 3-level models. Within-patient coefficients were unbiased in both 2-and 3-level models. In 3-level models, standard errors were biased when number of therapists was small, and this bias increased in unbalanced designs. Ignoring random slopes led to biased standard errors when slope variance was large; but 2-level models still outperformed 3-level models. Conclusions: In contrast to treatment outcome research, when studying mechanisms of change on a within-patient level, modeling therapist effects may even reduce model performance and increase bias.

Clients’ perceptions of the working alliance as a predictor of increases in positive affect.

“…We confirmed our hypothesis, showing that increases in participants' perceptions of the alliance with their therapists predicted increases in their positive affect over the course of therapy. Importantly, we confirmed this hypothesis using two robust statistical models: (a) a multilevel model that included a covariate, allowed us to consider within-and betweenparticipants effects, and afforded us the opportunity to account for all data as well as the random intercept of client identification, and (b) a cross-lagged panel model using SEM via the xtdpdml command (Williams et al, 2018) that aligns with recent calls for cross-lagged models in psychotherapy research (Falkenström et al, 2020) and current best practices with cross-lagged panel models (i.e., use of SEM; Allison et al, 2017;Hamaker et al, 2015). It appears, drawn from these correlational data, that increases in alliance tend to coexist with increases in positive affect, even when psychological distress and session length are controlled.…”

“…We use this model not to assert causality, as cautioned against in the cross-lagged panel model literature (Hamaker et al, 2015), but to better understand the novel relationship suggested in our hypothesis. Further, this approach is advocated for in psychotherapy research (e.g., Falkenström et al, 2020) especially in the context of alliance research (e.g., Falkenström et al, 2016). Using the xtdpdml command in Stata (Williams et al, 2018) afforded us an opportunity to efficiently model the lagged relationship between alliance and positive affect.…”

Reduction in depressive symptoms predicts improvement in eating disorder symptoms in interpersonal psychotherapy: results from a naturalistic study

“…For within-person coefficients, statistical power and coefficient bias are determined roughly by the product of these dimensions. A recent simulation study [50] on a similar SEM model as the one used in the present study showed that with N = 50 and T = 5, i.e. 250 observations in total, statistical power was well above 80% for finding a medium-sized effect, and average coefficient bias was negligible.…”

Negative mood regulation as a mechanism of change in cognitive therapy for depression.

“…Moreover, in our study we did not include therapist effects as a third level in the model. However, as noted above recent simulation studies have shown that including therapist effect as a third level in these models, may represent a source of bias, in the context of an unbalanced design (i.e., when the number of patients per therapist is not constant; Falkenström et al, 2020).…”