Although it has been well-documented that older adults spend a significant amount of time being sedentary and have slower gait velocity, little is known of how physical fitness mediates the association between them. The main purpose of this study was to explore whether objectively measured physical fitness mediates the association between domain-specific and total sedentary behavior and gait velocity. We recruited 120 older adults aged ≥ 60 years. Sedentary behavior was assessed by the Measure of Older Adults’ Sedentary Time questionnaire. We used a Zebris pressure platform to assess gait velocity. To assess the level of overall physical fitness, we summed the z-scores of seven tests: (1) waist circumference, (2) chair stand in 30 s, (3) arm curl in 30 s, (4) 2-min step test, (5) chair sit-and-reach test, (6) back scratch test, and (7) 8-foot up-and-go test. Overall physical fitness was obtained by summing up all physical test z-scores. Gait velocity was significantly associated with all domain-specific and total sedentary behavior (β = −0.04 to −0.35, p < 0.05). Overall physical fitness was significantly associated with all domain-specific and total sedentary behavior (β = −0.21 to −1.24, p < 0.001) and gait velocity (β = 0.23 to 0.24, p < 0.001). When physical fitness was put as the mediator, significant direct effects between sedentary behavior and gait velocity disappeared. Results indicate that physical fitness fully mediates the association between sedentary behavior and gait velocity in older adults.