Background: Epidemiologic evidence suggests that physical activity benefits cognition, but results from randomized trials in sedentary individuals are limited and inconsistent. Objective: To evaluate the effects of physical activity on cognition among sedentary older adults. Objective: A systematic literature search for eligible studies published up to January 1, 2021, was performed on six international (PubMed, Cochrane Library, Web of Science, Sinomed, FMRS, and OVID) and three Chinese databases (Wanfang, China National Knowledge Infrastructure, and VIP). We estimated the effect of physical activity on the cognition of sedentary elderly by standardized mean differences (SMD) and 95% confidence intervals (CI) using a random-effects model. We evaluated publication bias using funnel plots and heterogeneity using I2 statistics. Subgroup analyses were conducted by baseline cognition, intervention duration, activity type, and country. Results: Seven randomized controlled trials (RCTs) comprising 321 (experimental group, 164; control group, 157) sedentary older adults were included in the meta-analysis. Physical activity significantly improved cognition in sedentary elderly adults compared with controls (SMD: 0.50, 95% CI:0.09–0.92). Subgroup analyses showed significant effects of baseline cognition impairment (SMD: 9.80, 95% CI: 5.81–13.80), intervention duration > 12 weeks (SMD: 2.85, 95% CI: 0.73–4.96), aerobic exercise (SMD: 0.74, CI: 0.19–1.29), and countries other than the United States (SMD: 10.50, 95% CI: 7.08–13.92). Conclusion: Physical activity might have a general positive effect on the cognition of sedentary older adults. Intervention > 12 weeks and aerobic exercise can effectively delay their cognitive decline; however, more rigorous RCTs are needed to support our findings.