A method for making inferences in network analysis: Comment on Forbes, Wright, Markon, and Krueger (2017).

Abstract

Forbes, Wright, Markon, and Krueger (2017) make a compelling case for proceeding cautiously with respect to the overinterpretation and dissemination of results using the increasingly popular approach of creating “networks” from co-occurrences of psychopathology symptoms. We commend the authors on their initial investigation and their utilization of cross-validation techniques in an effort to capture the stability of a variety of network estimation methods. Such techniques get at the heart of establishing “reproducibility”, an increasing focus of concern in both psychology (e.g., Pashler & Wegenmakers, 2012) and science more generally (e.g., Baker, 2016). However, as we will show, the problem is likely worse (or at least more complicated) than they initially indicated. Specifically, for multivariate binary data, the marginal distributions enforce a large degree of structure on the data. We show that some expected measurements – such as commonly used centrality statistics – can have substantially higher values than what would usually be expected. As such, we propose a nonparametric approach to generate confidence intervals through Monte Carlo simulation. We apply the proposed methodology to the National Comorbidity Survey - Replication, provided by Forbes et al, finding that the many of the results are indistinguishable from what would be expected by chance. Further, we discuss the problem of multiple testing and potential issues of applying methods developed for one-mode networks (e.g., ties within a single set of observations) to two-mode networks (e.g., ties between two distinct sets of entities). When taken together, these issues indicate that the psychometric network models should be employed with extreme caution and interpreted guardedly.

Core psychopathology in anorexia nervosa and bulimia nervosa: A network analysis

“…Moreover, concerns have recently been raised that network analyses have limited replicability (Forbes, Wright, Markon, & Kreuger, ; though see Borsboom et al, ; Epskamp, Borsboom, & Fried, ). In fact, two research groups concluded that current network estimation methods produce unreliable results with “poor and absent replicability” (Forbes et al, , p. 981; Steinley, Hoffman, Brusco, & Sher, ). These critiques highlight the need for high‐quality ED network replication studies, as unique characteristics about published ED networks raise replicability concerns.…”

Further evidence that psychopathology networks have limited replicability and utility: Response to Borsboom et al. (2017) and Steinley et al. (2017)

“…While network theory (e.g., Borsboom, 2017) calls attention to interesting and potentially important avenues for future research, the current psychopathology network methodologies are plagued with substantial flaws. For example, in the few months since our target article was accepted for publication, a series of papers have emerged on substantial problems facing the methods, in addition to poor replicability (Bos et al, 2017;Epskamp et al, 2017;Fried & Cramer, in press;Guloksuz, Pries & van Os, 2017;Steinley et al, 2017;Wichers, Wigman, Bringmann & de Jonge, 2017). Taken together, this growing literature highlights the remarkable sensitivity of psychopathology network results to a multitude of factors including the study design, variables included, characteristics of the data, and analytic methods (e.g., Bulteel, Tuerlinckx, Brose & Ceulemans, 2016;Terluin, de Boer & de Vet, 2016;Wichers et al, 2017).…”

“…The two commentaries on our target article (Forbes, Wright, Markon, & Krueger, 2017), from advocates of psychopathology networks and scholars versed in multivariate statistical methodology (Steinley, Hoffman, Brusco & Sher, 2017) differ substantially in their assessment of our findings. Borsboom et al question the accuracy and validity of our work, and present a re-analysis of the data that they suggest provides evidence that "network models replicate very well" (p. 3).…”

Replicating latent structural models of psychopathology: Reply to Kotov, Ruggero, Krueger, Watson, and Zimmerman (2018)

“…Steinley, Hoffman, Brusco, & Sher, 2017). Much of the research on the latent structure of psychopathology relies on dichotomous diagnosis-level data.…”

“…As with other binary data analysis contexts, measures of association among diagnoses are sensitive to base rates, which vary widely across community, outpatient, and inpatient settings; for example, relationships between binary disorders are often highly inflated by noncases because of the low prevalence of disorders in unselected, community-based samples (cf. Steinley, Hoffman, Brusco, & Sher, 2017).…”