Purpose: This cross-sectional study assessed the associations of gender, age, level of intellectual disabilities (IDs) and of daily sedentary and physical activity (PA) time with physical fitness in adults with ID. Materials and methods: Sixty adults (mean age = 39.19 ± 11.70 years) with ID participated in this cross-sectional study. PA was monitored for 7 days using an ActiGraph GT3X monitor. Physical fitness was measured with a 6-min walking test, isometric push-up test, modified curl-up test, handgrip strength test, and back-saver sit-and-reach test. Results: (a) An age of ≥39 years and female gender were associated with lower performance in multiple aspects of physical fitness. (b) More moderate-to-vigorous PA (MVPA) was associated with greater muscular strength and endurance (modified curl-ups: β = 0.36, p < 0.01; handgrip strength: right, β = 0.52, p < 0.01; left, β = 0.52, p < 0.01). (c) More light-intensity PA (LPA) was associated with greater upper-body muscular endurance (β = 0.42, p < 0.01) and greater flexibility (right leg: β = 0.36, p < 0.01; left leg: β = 0.38, p < 0.01). Conclusion: LPA may be as beneficial as MVPA to the physical fitness of adults with ID. Future studies should focus on developing effective PA interventions for adults with ID, especially for women and individuals aged ≥39 years, by incorporating both LPA and MVPA.