Bipolar disorder (BD) and major depressive disorder (MDD) share similar clinical characteristics that often obscure the diagnostic distinctions between these conditions. Both functional and structural brain abnormalities have been reported in these two disorders. However, the direct link between altered functioning and structure in these two diseases is unknown. To elucidate this relationship, we conducted a multimodal fusion analysis on the functional network connectivity (FNC) and gray matter density (GMD) from MRI data from 13 BD, 40 MDD, and 33 matched healthy controls (HC). A data-driven fusion method called mCCA+jICA was used to identify the co-altered FNC and GMD components. We found reduced GMD in the parietal and occipital cortices related to lower FNC in a sensory-motor network as well as stronger interconnection in frontal regions in BD compared to HC. Meanwhile, the MDD group exhibited GMD deficits in the amygdala and cerebellum. Among preliminary classifiers trained using features generated from these group discriminative components, the overall accuracy with resampling and cross-validation reached 91.3% for 3 groups and 99.5% for BD versus MDD. Our findings suggest that both overlapping and unique functional and structural deficits exist in BD and MDD, and the abnormalities may be utilized as potential diagnostic biomarkers.