Our objective in this paper was to systematically review evaluations of the effects of exercises on pain symptoms and activities of daily living (ADL) in middle-aged and elderly patients with low back pain (LBP). We searched Web of Science, PubMed, EBSCO, and China National Knowledge Internet (CNKI) databases for randomized controlled trials (RCTs) on this topic. We evaluated the methodological quality of included articles using the Physiotherapy Evidence Database (PEDro) scale, and we statistically analyzed these studies using RevMan software. We reviewed 18 RCTs (23 comparison groups) with a total of 910 participants, and our meta-analysis confirmed that exercises significantly improved both pain and ADLs measured on visual analog scales (VAS) (SMD = −0.91, 95% CI: [−1.3, −0.52], p < 0.00001) and on the Oswestry Disability Index (ODI) (SMD = −2.07, 95% CI: [−3.19, −0.96], p < 0.00001). We conclude that exercises can reduce pain severity and improve ADL capacity in middle-aged and elderly persons with LBP, confirming that exercise can serve as a medical intervention for these indivdiuals. However, given the high heterogeneity of responses among individual participants, there remains a need for further study.