Background and Objectives: Stoma surgery is linked to reduced physical activity in colon cancer survivors and leads to decreased physical fitness, activity of daily living dysfunction, and poorer quality of life. However, few studies have investigated the physical fitness levels of colon cancer survivors living with stomas. This study aimed to compare the physical fitness levels of colon cancer survivors with stomas and healthy adults, assessing them in a variety of dimensions (e.g., strength and flexibility) and in terms of physical fitness age (PFA), an integrated index of overall fitness. Materials and Methods: The study population consisted of 17 colon cancer survivors with (stoma group) and 20 healthy adults without (control group) a stoma. Physical fitness was assessed using a battery of five tests: repeated back-and-forth steps, 30-s chair stand, chair sit-and-reach, grip strength, and single-leg balance with eyes closed. Respective performance values were converted into PFA, which was compared between the stoma and control groups. Fitness indicators were compared between groups by analysis of covariance, and PFA and chronological age (CA) by paired t-tests. Results: The mean ages (±standard deviation) of the stoma and control groups were 74.1 ± 7.9 and 73.5 ± 7.1 years, respectively. Colon cancer survivors with stomas had poorer lower limb muscular strength, endurance, and flexibility than controls. In the stoma group, the marginal mean (±standard error) PFA was calculated to be 82.5 ± 3.7 years, significantly higher than the CA and PFA of the control group (69.6 ± 3.9 years). Conclusions: Colon cancer survivors with stomas have lower physical fitness levels than healthy adults, with apparent deficits in lower limb flexibility, muscular strength, and endurance. Our findings demonstrated the need for exercise interventions in this population, focusing on these dimensions of fitness. However, our results should be corroborated by means of a larger-scale comparison in future studies.