Neuroimaging techniques have been increasingly used to understand the neural biology of aging brains. The neuroimaging variables from distinct brain locations and modalities can exhibit age-related patterns that reflect localized neural decline. However, it is a challenge to identify the impacts of risk factors (eg, mental disorders) on multivariate imaging variables while simultaneously accounting for the dependence structure and nonlinear age trajectories using existing tools. We propose a mixed-effects model to address this challenge by building random effects based on the latent brain aging status. We develop computationally efficient algorithms to estimate the parameters of new random effects. The simulations show that our approach provides accurate parameter estimates, improves the inference efficiency, and reduces the root mean square error compared to existing methods. We further apply this method to the UK Biobank data to investigate the effects of tobacco smoking on the white matter integrity of the entire brain during aging and identify the adverse effects on white matter integrity with multiple fiber tracts.