This study aimed to examine the association of alcohol and tobacco use with severity of depression in older age. Analyses were performed on a pooled data set (n = 3724) from two German old-age cohort studies (LEILA 75+, 6 follow-ups and AgeCoDe/AgeQualiDe, 9 follow-ups). Depressive symptoms were assessed via two screening scales for depression (CES-D and GDS-15) which were harmonized for pooled analysis. A mixed-effects linear regression model for the total sample and additional stratified models for men and women were used. Smoking at baseline was significantly associated with a higher level of depression severity (β = 0.142, 95% CI: 0.051–0.233, p = 0.002), whereas drinking was significantly associated with a decreased level of depression (β = −0.069, 95% CI: −0.119–−0.021, p = 0.005). Concurrent substance use at baseline increased longitudinal depression severity (β = 0.193, 95% CI: −0.011–0.375, p = 0.037). Analyses stratified by gender showed a significant inverse association between drinking and depressive symptoms in men (β = −0.138, 95% CI: −0.231–−0.045, p = 0.004), but not in women (β = −0.060, 95% CI: −0.120–0.001, p = 0.052). Given the burden of major depression, it is important that health care providers, especially primary care physicians, assess and monitor lifestyle factors, even at older ages.