(1) Background: The present study aimed to investigate the association between body mass index (BMI) and muscular fitness in Chinese college freshmen. (2) Methods: A total of 6425 college freshmen in mainland China were recruited. BMI was classified as underweight (<18.5 kg/m2), normal weight (18.5~23.9 kg/m2), overweight (24~27.9 kg/m2), and obese (≥28 kg/m2), according to the Working Group on Obesity in China. Health-related physical fitness components including cardiorespiratory fitness, lower body explosive power, upper body muscular endurance, abdominal muscular endurance, flexibility, and vital capacity were assessed. Physical fitness index and muscular fitness index were calculated, respectively, as the sum score of the standardized values (z-score) of the corresponding components. Three regression models were used to evaluate the potential associations: a linear regression model, a polynomial regression model, and a restricted cubic spline regression model. Adjust R square was used to compare among models. (3) Results: Significant differences were observed among different BMI categories in nearly all physical fitness components as well as physical fitness z-score and muscular fitness z-score (p < 0.001), regardless of gender. Significant linear associations were found between BMI and physical fitness z-score as well as muscular fitness z-score among total, male, and female groups (p < 0.05). However, the restricted cubic spline regression model showed a better fitting effect (adjust R2 was 7.9%, 11.2%, and 4.8% in total, male, and female for physical fitness and 7.7%, 15.7%, and 4.0%, for muscular fitness, respectively), compared with the linear and polynomial regression models, presented by a higher adjusted R2. Restricted cubic splines analysis showed that BMI value and physical fitness z-score showed a non-linear relationship with an approximate inverted U curve in all groups, while an approximate reversed J-shaped association was observed between BMI and muscular fitness z-score in all groups. (4) Conclusions: The present study showed a nonlinear negative relationship between BMI and physical fitness with underweight and overweight/obese college freshmen having poorer physical fitness and muscular fitness than their normal BMI peers, which may provide useful evidence to the development of public health recommendations and encourage the health management of young adults. Future studies should further explore the relationship between BMI and muscular fitness with multi-centered large sample size studies.