The aging population routinely has comorbid conditions requiring complicated medication regimens, yet non-adherence can preclude optimal outcomes. This study explored the association of adherence in the elderly with demographic, socio-economic and disease burden measures. Data were from the fifth visit (2011–2013) for 6,538 participants in the Atherosclerosis Risk in Communities Study, conducted in four communities. The Morisky Green Levine Scale measured self-reported adherence. Forty percent of respondents indicated some non-adherence, primarily due to poor memory. Logit regression showed, surprisingly, that persons with low reading ability were more likely to report being adherent. Better self-reported physical or mental health both predicted better adherence, but the magnitude of the association was greater for mental than for physical health. Compared to persons with normal or severely impaired cognition, mild cognitive impairment was associated with lower adherence. Attention to mental health measures in clinical settings could provide opportunities for improving medication adherence.