Stability test of canonical correlation analysis for studying brain‐behavior relationships: The effects of subject‐to‐variable ratios and correlation strengths

Abstract

Canonical correlation analysis (CCA), a multivariate approach to identifying correlations between two sets of variables, is becoming increasingly popular in neuroimaging studies on brain-behavior relationships. However, the CCA stability in neuroimaging applications has not been systematically investigated. Although it is known that the number of subjects should be greater than the number of variables due to the curse of dimensionality, it is unclear at what subject-to-variable ratios (SVR) and at what correlation strengths the CCA stability can be maintained. Here, we systematically assessed the CCA stability, in the context of investigating the relationship between the brain structural/functional imaging measures and the behavioral measures, by measuring the similarity of the first-mode canonical variables across randomly sampled subgroups of subjects from a large set of 936 healthy subjects. Specifically, we tested how the CCA stability changes with SVR under two different brain-behavior correlation strengths. The same tests were repeated using an independent data set (n = 700) for validation. The results confirmed that both SVR and correlation strength affect greatly the CCA stability-the CCA stability cannot be guaranteed if the SVR is not sufficiently high or the brain-behavior relationship is not sufficiently strong. Based on our quantitative characterization of CCA stability, we provided a practical guideline to help correct interpretation of CCA results and proper applications of CCA in neuroimaging studies on brain-behavior relationships.

Dysconnection and cognition in schizophrenia: a spectral dynamic causal modeling study

“…Working memory was negatively related to the other two domains. Notably, this linear relationship was achieved with a sample to feature ratio of 58/(3+3) ≈ 9.7, in line with recent recommendations for multivariate model stability (Helmer et al, 2020;Yang et al, 2021).…”

“…Cross-validation confirmed that such a linear association has some level of out-of-sample generalizability. To achieve a stable and generalizable CCA model, we had observed the sample to feature ratio of above nine (Helmer et al, 2020; Yang et al, 2021) using ensemble feature selection (Bin et al, 2016; Saeys et al, 2008). We shall now discuss the significance of the effective connections and cognitive domains that the feature selection procedure returned—and their contribution towards the connection-cognition model.…”

“…Based on previous rigorous studies (Helmer et al, 2020; Leach and Henson, 2014; Yang et al, 2021), constructing reliable CCA models requires that the number of samples be at least nine to ten times the number of features. In the current dataset, cognitive scores were available for a total of 117 subjects, whereas there were 470 EC features.…”

“…This selection procedure continues until the step at which the minimum partial Λ exceeds a predetermined threshold, or equivalently the associated partial F falls below a preselected value 15 . In practice, to the keep the sample to feature ratio above nine (Helmer et al, 2020; Leach and Henson, 2014; Yang et al, 2021), the feature selection was stopped earlier.…”

“…In order to establish functional validity for the EC estimates, and to elucidate the neural associates of cognitive impairment in SZ, we conducted a canonical correlation analysis (CCA) between the EC parameters and the cognitive scores of the patients (from MCCB 2 tests). To achieve a stable CCA model, we observed the practical recommendations (about sufficient sample to feature ratio) from recent technical reports (Helmer et al, 2020; Yang et al, 2021) using appropriate feature selection procedures. From this analysis, we asked which effective connections and cognitive traits are mostly correlated in the patients, and whether such a linear association holds some degree of generalizability beyond the current sample.…”