Background Major depressive disorder and bipolar disorder in adolescents are prevalent and are associated with cognitive impairment, executive dysfunction, and increased mortality. Early intervention in the initial stages of major depressive disorder and bipolar disorder can significantly improve personal health. Methods We collected 309 samples from the Adolescent Brain Cognitive Development study, including 116 adolescents with bipolar disorder, 64 adolescents with major depressive disorder, and 129 healthy adolescents, and employed a support vector machine to develop classification models for identification. We developed a multimodal model, which combined functional connectivity of resting-state functional magnetic resonance imaging and four anatomical measures of structural magnetic resonance imaging (cortical thickness, area, volume, and sulcal depth). We measured the performances of both multimodal and single modality classifiers. Results The multimodal classifiers showed outstanding performance compared with all five single modalities, and they are 100% for major depressive disorder versus healthy controls, 100% for bipolar disorder versus healthy control, 98.5% (95% CI: 95.4–100%) for major depressive disorder versus bipolar disorder, 100% for major depressive disorder versus depressed bipolar disorder and the leave-one-site-out analysis results are 77.4%, 63.3%, 79.4%, and 81.7%, separately. Conclusions The study shows that multimodal classifiers show high classification performances. Moreover, cuneus may be a potential biomarker to differentiate major depressive disorder, bipolar disorder, and healthy adolescents. Overall, this study can form multimodal diagnostic prediction workflows for clinically feasible to make more precise diagnose at the early stage and potentially reduce loss of personal pain and public society.