Few studies have examined the effects of widowhood on cognitive function in Chinese elderly individuals. We conducted a longitudinal study to assess the association between widowhood and cognitive function and further explored gender differences in this association and the impact of widowhood duration. The analytical sample consisted of 5872 Chinese elderly people who participated in the Chinese Longitudinal Healthy Longevity Survey (CLHLS) and were followed up from 2005 to 2014. We used the Chinese version of the Mini-Mental State Examination (MMSE) to assess cognitive function. Widowhood duration was calculated from the self-reported year at which the spouse passed away. Multilevel growth models were employed to estimate the association between widowhood and cognitive function while adjusting for many demographic and socioeconomic characteristics. Widowhood status was associated with cognitive decline among Chinese elderly individuals after adjusting for covariates (B = −0.440, 95% CI −0.727 to −0.152), and this association was only statistically significant among men (B = −0.722, 95% CI −1.104 to −0.339). Being widowed for 5 years or less (B = −0.606, 95% CI −1.112 to −0.100), 16–20 years (B = −0.937, 95% CI −1.685 to −0.190), and 21+ years (B = −1.401, 95% CI −1.967 to −0.834) predicted worse cognitive function in men, while being widowed for more than 21+ years (B = −0.655, 95% CI −1.186 to −0.124) was associated with cognitive decline in women. More attention should be directed towards widowed men and long-term widowed elderly individuals.