Based on the data of four periods of CLHLS (2008, 2011, 2014, 2018), the latent variable growth model (LGCM) was applied to 2344 older adults who completed four follow-up surveys, to study the trajectory of leisure activities and cognitive ability and explore the relationship between leisure activities and cognitive ability of older adults. The results showed that: (1) leisure activities and cognitive ability of older adults showed a non-linear downward trend; (2) leisure activities significantly and positively predicted the cognitive ability of older adults at every time point; (3) the initial level of leisure activity positively predicted the initial level of cognitive ability but negatively predicted the rate of cognitive decline; In addition, cognitive activities had a greater effect on cognitive ability than non-exercise physical activities; (4) the rate of decline of leisure activities also significantly and positively predicted the rate of decline of cognitive ability; (5) cross-lagged regression analysis further suggested the overall positive predictive effect of leisure activity on cognitive ability; (6) overall, education level had a significant contribution to cognitive ability, and the higher the education level, the slower the decline of cognitive ability; and (7) smoking could promote cognitive ability in older adults and no significant effect was found between alcohol drinking and cognitive ability. Accordingly, the government should encourage older adults to do more leisure activities, especially the cognitive activity, to effectively prevent cognitive decline.