Using a longitudinal approach, we sought to define the interplay between genetic and nongenetic factors in shaping vulnerability or resilience to COVID-19 pandemic stress, as indexed by the emergence of symptoms of depression and/or anxiety. University of Michigan freshmen were characterized at baseline using multiple psychological instruments. Subjects were genotyped, and a polygenic risk score for depression (MDD-PRS) was calculated. Daily physical activity and sleep were captured. Subjects were sampled at multiple time points throughout the freshman year on clinical rating scales, including GAD-7 and PHQ-9 for anxiety and depression, respectively. Two cohorts (2019 to 2021) were compared to a pre-COVID-19 cohort to assess the impact of the pandemic. Across cohorts, 26 to 40% of freshmen developed symptoms of anxiety or depression (N = 331). Depression symptoms significantly increased in the pandemic years and became more chronic, especially in females. Physical activity was reduced, and sleep was increased by the pandemic, and this correlated with the emergence of mood symptoms. While low MDD-PRS predicted lower risk for depression during a typical freshman year, this genetic advantage vanished during the pandemic. Indeed, females with lower genetic risk accounted for the majority of the pandemic-induced rise in depression. We developed a model that explained approximately half of the variance in follow-up depression scores based on psychological trait and state characteristics at baseline and contributed to resilience in genetically vulnerable subjects. We discuss the concept of multiple types of resilience, and the interplay between genetic, sex, and psychological factors in shaping the affective response to different types of stressors.