Introduction We explored how different chronic diseases, risk factors, and protective factors highly associated with cardiovascular diseases (CVD) are associated with dementia or Mild Cognitive Impairment (MCI) in Korean elders, with a focus on those that manifest in mid-life. Methods A CVD-free cohort (n = 4289) from the Korean Longitudinal Study of Aging was selected to perform Cox mixed-effects proportional hazard regressions. Eighteen control variables with strong associations to CVD were chosen as explanatory variables, and Mini-Mental State Examination (MMSE) score cut-off for dementia and MCI were used as outcome variables. Results The statistically significant (P < 0.05) adverse factors that contribute in developing dementia were age (aHR 1.07, 1.05–1.09), Centre for Epidemiological Studies Depression Scale (CESD-10) (aHR 1.17, 1.12–1.23), diagnosis with cerebrovascular disease (aHR 3.73, 1.81–7.66), living with diabetes (aHR 2.30, 1.22–4.35), and living with high blood pressure (HBP) (aHR 2.05, 1.09–3.87). In contrast, the statistically significant protective factors against developing dementia were current alcohol consumption (aHR 0.67, 0.46–0.99), higher educational attainment (aHR 0.36, 0.26–0.56), and regular exercise (aHR 0.37, 0.26–0.51). The factors with a statistically significant adverse association with progression to MCI were age (aHR 1.02, 1.01–1.03) and CESD-10 (aHR 1.17, 1.14–1.19). In contrast, the statistically significant protective factors against developing MCI were BMI (aHR 0.96, 0.94–0.98), higher educational attainment (aHR 0.33, 0.26–0.43), and regular exercise (aHR 0.83, 0.74–0.92). Conclusion In lieu of the protective factor of MCI and dementia, implementing regular exercise routine well before mid-life and cognitive decline is significant, with adjustments made for those suffering from health conditions, so they can continue exercising despite their morbidity. Further attention in diabetes care and management is needed for patients who already show decline in cognitive ability as it is likely that their MCI impacts their ability to manage their existing chronic conditions, which may adversely affect their cognitive ability furthermore.