Background: This study explored the effects of physical activity and sedentary behaviour on the decline of cognitive ability among the elderly. To compensate for the limitations of self-reported physical activity, objective measures were used. Methods: A cross-sectional survey of 308 aged people mean 68.66 ± 5.377 years, in Nanjing, China, was conducted. Physical activity was measured using the ActiGraph GT3X+, and cognitive function was measured using the Montreal Cognitive Assessment. Results: The overall participant model, adjusted for age, BMI, education, and monthly average income, found that light physical activity (β = 0.006, p < 0.01), moderate-vigorous physical activity (β = 0.068, p < 0.001), and total physical activity (β = 0.006, p < 0.01) had a significant linear relationship with cognitive ability, while sedentary time did not (β = − 0.020, p>0.05). Further, light physical activity only affects the cognitive ability of elderly females (β = 0.006, p < 0.05). There was an inverted 'U' association between moderate-vigorous physical activity and cognitive ability. The association models found that moderate-vigorous physical activity in the 22.13 min·day − 1~3 8.79 min·day − 1 range affected cognitive ability most beneficially, with the highest beta coefficient among all groups (β = 0.091, p < 0.05). Conclusions: While physical activity can significantly improve cognitive ability among the elderly, sedentary behaviour is associated with decreased cognitive function across genders.