The relationship between prediabetes and dementia remains controversial. We aimed to examine the association between cumulative exposure to impaired fasting glucose (IFG) and the risk of dementia in the general population. 1,463,066 middle-aged and elderly subjects who had had health examinations for four consecutive years were identified from a Korean nationwide population-based cohort database. IFG was defined as fasting blood glucose 100–125 mg/dL, and the risk of dementia—according to the number of IFG exposure (range 0–4)—was analyzed using the multivariable Cox proportional-hazards model. During the median 6.4 years of follow-up, 7,614 cases of all-cause dementia, 5,603 cases of Alzheimer’s disease, and 1,257 cases of vascular dementia occurred. There was a significant trend towards a higher risk of all-cause dementia (P for trend = 0.014) and Alzheimer’s disease (P for trend = 0.005) according to the cumulative exposure to IFG, but with a modest (approximately 7 to 14%) increase in the hazards. A significant stepwise increase in the risk of all-cause dementia and Alzheimer’s disease was seen in non-obese subjects, whereas no significant association was observed in obese subjects. This study supports the association between prediabetes and incident dementia and emphasizes that even mild hyperglycemia should not be overlooked.