The association of the degree of aging based on the whole-brain anatomical characteristics, or brain age, with smoking, alcohol consumption, and individual genetic variants is unclear. Here, we investigated these associations through analyzing data collected for UK Biobank subjects with an age range of 45 to 79 years old. We first trained a statistical model for obtaining relative brain age (RBA), a metric describing a subject's brain age relative to peers, based on a randomly selected training set subjects (n=2,679). We then applied this model to the evaluation set subjects (n=6,252) and further tested the association of RBA with tobacco smoking, alcohol consumption, and 529,098 genetic variants. We found that daily or almost daily consumption of smoking or alcohol was significantly associated with increased RBA (P<0.05). Interestingly, there was no significant difference of RBA among subjects who smoked occasionally, only tried once or twice, or abstained from smoking. Further, there was no significant difference of RBA among subjects who consumed alcohol 1 to 3 times a month, at special occasions only, or abstained from alcohol consumption. Among the subjects who smoked on most or all days and did not abstain from alcohol drinking, RBA increased by 0.021 years for each addition pack-year of smoking (P<0.05) and by 0.014 years for each additional gram of alcohol consumed (P<0.05). We did not identify individual genetic variation significantly associate with RBA. Further exploration of genetic variation-brain aging association is warranted, where our current genetic association statistics may serve as prior knowledge.