Grandchild caregiving is suggested to improve the elderly’s cognitive function, but the specific relationship remains under-investigated. Considering gender disparity, this study aimed to understand the relationship between grandchild caregiving and cognition. In total, 7236 Chinese residents (≥45 years old) were selected from the 2015 China Health and Retirement Longitudinal Study (CHARLS). The China Health and Retirement Longitudinal Study Harmonized Cognitive Assessment Protocol (CHARLS-HCAP) was used to measure cognition. Grandparenting was measured from three dimensions: caregiving frequency, intensity, and the number of grandchildren cared for. The relationship was examined by multivariate linear regression, with age as a moderator. The results showed that the majority of respondents provided care to their grandchildren, especially grandmothers. Grandchild caregiving was positively associated with cognition (β = 0.686, 95% CI = 0.334–1.038), especially in the older-aged group. Moderate, not regular grandparenting, or caring for one grandchild was more positively associated with cognitive function. However, intensive and regular grandchild care was significantly associated with cognition only in men. No moderating effects of age were found in women. The study confirmed that moderate intensity and frequency of caregiving was related to better cognitive function in middle-aged and older Chinese population, whereas cultural context and gender differences could be considered when designing targeted policies.