Depressive symptoms may either be a risk factor or prodromal to dementia. Investigating this association in midlife may help clarify the role of depression in cognitive aging. We aimed to identify trajectories in depressive symptoms in early to mid-life and related cognitive and brain outcomes in midlife. This study includes 3944 Black and White participants (ages 26−45 years at baseline) from the Coronary Artery Risk Development in Young Adults (CARDIA) study with 20 years of follow-up. Depressive symptoms were assessed using the Center for Epidemiological Studies Depression scale at five time points over 20 years. Growth mixture modeling (GMM) was used to identify depressive symptom trajectories. Participants completed a neuropsychological battery 20 years after baseline, including the Digit Symbol Substitution Test (DSST), Rey-Auditory Verbal Learning Test (RAVLT), Stroop Test, Montreal Cognitive Assessment (MoCA), and category and letter fluency tests. A sub-sample of participants (n = 662) underwent brain magnetic resonance imaging (MRI) to characterize gray matter volumes and white matter hyperintensities (WMHs). We identified four classes of depressive symptom trajectories: a “declining” class (n = 286, 7.3%) with initially high symptoms and subsequent decline, a class with consistently high symptoms (“steady high”; n = 264, 6.7%), a class with late increases in symptoms (“increasing”; n = 277, 7%), and a class with consistently low symptoms (“steady low”; n = 3117, 79.0%). The steady high and the increasing classes had poorer performance on all cognitive tests, while the declining class had poorer performance on the DSST, verbal fluency, and MoCA. Compared to the steady low symptom class, the steady high class had lower volumes in the entorhinal cortex (β: −180.80, 95% CI: −336.69 to −24.91) and the amygdala (β: −40.97, 95% CI: −74.09 to −7.85), the increasing class had more WMHs (β: 0.55, 95% CI: 0.22 to 0.89), and the declining class was not significantly different in any brain measures. Trajectories in depressive symptoms in young to mid-adulthood show distinct cognitive and brain phenotypes in midlife. Steady high depressive symptoms may represent a group that is at risk for dementia, whereas increasing symptoms in midlife may be associated with white matter damage.