History of incarceration is associated with an excess of morbidity and mortality. While the incarceration experience itself comes with substantive health risks (e.g., injury, psychological stress, exposure to infectious disease), most inmates eventually return to the general population where they will be diagnosed with the same age-related conditions that drive mortality in the non-incarcerated population but at exaggerated rates. However, the interplay between history of incarceration as a risk factor and more traditional risk factors for age-related diseases (e.g., genetic risk factors) has not been studied. Here, we focus on cognitive impairment, a hallmark of neurodegenerative conditions like Alzheimer’s disease, as an age-related state that may be uniquely impacted by the confluence of environmental stressors (e.g., incarceration) and genetic risk factors. Using data from the Health and Retirement Study, we found that incarceration and APOE-ϵ4 genotype (i.e., the chief genetic risk factor for Alzheimer’s disease) both constituted substantive risk factors for cognitive impairment in terms of overall risk and earlier onset. The observed effects were mutually independent, however, suggesting that the risk conveyed by incarceration and APOE-ϵ4 genotype operate across different risk pathways. Our results have implications for the study of criminal justice contact as a public health risk factor for age-related, neurodegenerative conditions.