Aim: To explore associations between sleep duration and physical fitness (PF) of children aged 3–6 years. Methods: This study investigated the sleep duration and PF data of children aged 3–6 years by stratified random sampling. The restricted cubic spline model and binary logistic regression analysis were mainly used for the empirical analysis of the correlation effect between sleep duration and PF. The final data had a total of 21,857 children, of which 11,245 (51.45%) were boys and 10,612 (48.55%) were girls. Results: The PF level of the children in this study showed a relatively positive level (pass rate = 93.6%), and 19.7% of them had abnormal sleep duration; the results of the restricted cubic spline showed an inverted U-shaped association between the level of PF and the risk of abnormal sleep duration (X2 = 28.13, p < 0.0001). The results of logistic regression analysis showed that children with abnormal sleep duration were more likely to have a low PF, body morphology and motor ability levels at an OR (95% CI) of 1.077 (1.023–1.133), 1.077 (1.016–1.142) and 1.035 (1.08–1.062), respectively. The results of the bias correlation analysis showed varying degrees of correlation between sleep duration and various components of children’s PF. Conclusion: Insufficient or excessive amounts of sleep were significantly associated with PF in children, with abnormal sleep duration leading to reduced levels of PF and its components.