The deviation between chronological age and age predicted using brain MRI is a putative marker of overall brain health. Age prediction based on structural MRI data shows high accuracy in common brain disorders. However, brain aging is complex and heterogenous, both in terms of individual differences and the underlying biological processes. Here, we implemented a multimodal model to estimate brain age using different combinations of cortical area, thickness and sub‐cortical volumes, cortical and subcortical T1/T2‐weighted ratios, and cerebral blood flow (CBF) based on arterial spin labeling. For each of the 11 models we assessed the age prediction accuracy in healthy controls (HC, n = 750) and compared the obtained brain age gaps (BAGs) between age‐matched subsets of HC and patients with Alzheimer's disease (AD, n = 54), mild (MCI, n = 90) and subjective (SCI, n = 56) cognitive impairment, schizophrenia spectrum (SZ, n = 159) and bipolar disorder (BD, n = 135). We found highest age prediction accuracy in HC when integrating all modalities. Furthermore, two‐group case–control classifications revealed highest accuracy for AD using global T1‐weighted BAG, while MCI, SCI, BD and SZ showed strongest effects in CBF‐based BAGs. Combining multiple MRI modalities improves brain age prediction and reveals distinct deviations in patients with psychiatric and neurological disorders. The multimodal BAG was most accurate in predicting age in HC, while group differences between patients and HC were often larger for BAGs based on single modalities. These findings indicate that multidimensional neuroimaging of patients may provide a brain‐based mapping of overlapping and distinct pathophysiology in common disorders.