We aimed to examine whether the use of aspirin is associated with change in cognitive performance over time, and whether this association is modified by the cognitive stages. This study included a total of 1866 subjects, including 509 subjects with normal cognition (NC), 985 subjects with mild cognitive impairment (MCI), and 372 patients with Alzheimer’s disease (AD). In each group, we further categorized our subjects into two groups based on their aspirin using conditions: Aspirin users and non-aspirin users. Mini-Mental State Examination (MMSE) was the cognitive outcome. Linear mixed models were conducted to examine the longitudinal relationship between the use of aspirin and cognitive performance in each diagnostic group. In the cross-sectional analysis, there were no significant differences in MMSE scores between non-aspirin users and aspirin users in subjects with NC, subjects with MCI or patients with AD. In the longitudinal analysis, we detected an association of the baseline use of aspirin with cognitive decline (MMSE) over time in patients with AD, but not in the NC group or MCI group. Specifically, in AD patients, the use of aspirin at baseline was associated with slower cognitive decline over time. Our data may support an association between the use of aspirin and slower cognitive decline, while this association may be dependent on the clinical stages.