Assessing and Improving Robustness of Psychological Research Findings in Four Steps

Abstract

This chapter is accepted for publication, but still needs to undergo copy-editing. Cite as: Nuijten, M. B. (accepted chapter). Assessing and improving robustness of psychological research findings in four steps. In W. O'Donohue, A. Masuda, and S. Lilienfeld (Eds.), Clinical psychology and questionable research practices. Springer.

A constructive replication of client change during psychodynamic treatment in an outpatient setting

“…Replication can increase practitioners' confidence in applying findings when providing evidence-based treatment, with particular relevance to 'clinical decision-making' (Palmer et al, 2021, p. 361). In response to calls for more replications (e.g., Nuijten, 2022), we conducted a replication of Jankowski et al's (2019) naturalistic study on the effectiveness of long-term psychodynamic psychotherapy. Jankowski et al found subgroups of responders, clients for whom depression and anxiety symptoms reduced and life satisfaction and social functioning improved.…”

“…Consistent with a robustness check, our intent was to verify results 'through reanalysis before replicating a study in a new [independent] sample' (Nuijten, 2022, p. 381), and consistent with constructive replication, we examined whether findings were 'due to specific boundary conditions inherent to the original study' (Hüffmeier et al, 2016, p. 86). Furthermore, our application of three alternative analytic strategies to examine their influence on the substantive interpretations offered by Jankowski et al (2019) is a form of conceptual replication (Crandall & Sherman, 2016;Nuijten, 2022).…”

“…Ram and Grimm (2009, p. 572) described longitudinal mixture modelling as an ‘ exploratory … post‐hoc analysis technique that seeks out the story the data are trying to tell—a story that is limited by the specific bounds imposed during model specification’. More broadly, Flake and Fried (2020, p. 457) highlighted how ‘decisions made along the way can lead to fundamentally different conclusions’, and Nuijten (2022, p. 392) posed the question, ‘how sensitive is the result to different analytical approaches?’. Similarly, Shrout and Rodgers (2018, p. 504) suggested that replication could involve implementing ‘new statistical methodology for gaining insights from existing data’.…”

“…Similarly, Shrout and Rodgers (2018, p. 504) suggested that replication could involve implementing ‘new statistical methodology for gaining insights from existing data’. Replication varies along a continuum between direct and conceptual (Crandall & Sherman, 2016; Nuijten, 2022), and ‘unambiguously classify[ing] a given replication study’ can be challenging (Hüffmeier et al., 2016, p. 82). Consistent with a robustness check , our intent was to verify results ‘through reanalysis before replicating a study in a new [independent] sample’ (Nuijten, 2022, p. 381), and consistent with constructive replication , we examined whether findings were ‘due to specific boundary conditions inherent to the original study’ (Hüffmeier et al., 2016, p. 86).…”

“…Replication varies along a continuum between direct and conceptual (Crandall & Sherman, 2016; Nuijten, 2022), and ‘unambiguously classify[ing] a given replication study’ can be challenging (Hüffmeier et al., 2016, p. 82). Consistent with a robustness check , our intent was to verify results ‘through reanalysis before replicating a study in a new [independent] sample’ (Nuijten, 2022, p. 381), and consistent with constructive replication , we examined whether findings were ‘due to specific boundary conditions inherent to the original study’ (Hüffmeier et al., 2016, p. 86). Furthermore, our application of three alternative analytic strategies to examine their influence on the substantive interpretations offered by Jankowski et al.…”