Mounting evidence shows that the complex interaction between genes and environmental exposures is one of the major reasons for the diverse trajectories of micro- and macro-structural alterations across individuals. However, it is largely elusive how the multi-factorial mechanism, through which the gene-by-environment interaction exerts pathogenetic effects, contributes to brain structure differentially in male and female groups, in different age groups, and under different socioeconomic status (SES). To answer these important neuroscience questions, we performed a large-scale association analysis of brain structural measures and a set of risk factors of interest by using 35,921 subjects from UK Biobank (UKB), where each subject has 101 regional brain volumes, 110 white matter integrity measures, Apolipoprotein E (APOE) genotypes, 15 polygenic risk scores, demographic information, lifestyle factors, and socioeconomic status (SES). We found over 200 significant pairs of brain-wide associations (p < 2.37x10-4) regarding both the main and interaction effects by using exclusively white British UKB subjects, while validating most of them in an independent dataset of non-British UKB ancestry subjects. For example, a positive age-APOE-ε2 interaction effect on the amygdala volume was identified, suggesting its protective role of APOE-ε2 against brain aging specifically at the amygdala. Our findings allow us to establish an in-depth understanding of health disparities that may ensure that health care is accessible and equal for all, regardless of sex, race, ethnicity, geography, and socioeconomic status.