Gestational exposure to manganese (Mn), an essential trace element, is associated with fetal and childhood physical growth. However, it is unclear which period of growth is more significantly affected by prenatal Mn exposure. The current study was conducted to assess the associations of umbilical cord-blood Mn levels with birth outcomes and childhood continuous physical development. The umbilical cord-blood Mn concentrations of 1179 mother–infant pairs in the Sheyang mini birth cohort were measured by graphite furnace atomic absorption spectrometry (GFAAS). The association of cord-blood Mn concentrations with birth outcomes, and the BMI z-score at 1, 2, 3, 6, 7 and 8 years old, were estimated separately using generalized linear models. The relationship between prenatal Mn exposure and BMI z-score trajectory was assessed with generalized estimating equation models. The median of cord-blood Mn concentration was 29.25 μg/L. Significantly positive associations were observed between Mn exposure and ponderal index (β, regression coefficient = 0.065, 95% CI, confidence interval: 0.021, 0.109; p = 0.004). Mn exposure was negatively associated with the BMI z-score of children aged 1, 2, and 3 years (β = −0.383 to −0.249, p < 0.05), while no significant relationships were found between Mn exposure and the BMI z-score of children at the age of 6, 7, and 8 years. Prenatal Mn exposure was related to the childhood BMI z-score trajectory (β = −0.218, 95% CI: −0.416, −0.021; p = 0.030). These results indicated that prenatal Mn exposure was positively related to the ponderal index (PI), and negatively related to physical growth in childhood, which seemed most significant at an early stage.