Exposure to ambient air pollution and its threat to human health is a global concern, especially in the elderly population. Therefore, more in-depth studies are required to understand the extent of the harmful effects of particulate matter (PM) based on duration and levels of exposure. An investigation was conducted to determine the association between short- (1–14 days), medium- (1, 3, and 6 months), and long-term (1, 2, and 3 years) exposure to air pollutants (PM2.5 and PM10) and cognitive function among Koreans (4175 participants, mean age 67.8 years, 55.2% women) aged over 50 years. Higher levels of PM2.5 exposure for short to long term and PM10 exposure for medium to long term were found to be associated with decreased cognitive function, as indicated by lower scores of the Mini-Mental State Examination adopted in Korean (K-MMSE). There were significant effect modifications by sex, age group, alcohol consumption, physical activity, and smoking status in the association between long-term PM2.5 and PM10 exposure and cognitive function. These findings, which underscore the importance of the efforts to reduce the exposure levels and durations of air pollutants, especially in the vulnerable elderly population, provide evidence for establishing more stringent policies for air pollution regulations.