Brain age refers to age predicted by brain features. Brain age has previously been associated with various health and disease outcomes and suggested as a potential biomarker of general health. Few previous studies have systematically assessed brain age variability derived from single and multi-shell diffusion magnetic resonance imaging data. Here, we present multivariate models of brain age derived from various diffusion approaches and how they relate to bio-psycho-social variables within the domains of sociodemographic, cognitive, life-satisfaction, as well as health and lifestyle factors in midlife to old age (N = 35,749, 44.6 to 82.8 years of age). Bio-psycho-social factors could uniquely explain only a small proportion of the brain age variance. We observed a similar pattern across diffusion approaches. We consistently identified bio-psycho-social associations of interest across models. Waist-to-hip ratio, diabetes, hypertension, smoking, matrix puzzles solving, and job and health satisfaction and perception were associated with brain age. At the same time, sex and ethnicity showed large variability in their predictor strengths. As expected, chronological age contributed the strongest to explaining variability in brain age. Our results show that brain age cannot be sufficiently explained by bio-psycho-social variables alone. However, the observed associations suggest to adjust for sex, ethnicity, cognitive factors, as well as health and lifestyle factors.