Infant birth sizes are vital clinical parameters to predict poor growth and micronutrient deficiency in early life. However, their effects on childhood anemia remain unclear. We aimed to explore the associations between birth weight, crown–heel length, and head circumference with anemia in early childhood, as well as potential modification factors. This population-based prospective cohort study included 204,556 participants with singleton live births delivered at gestational ages of 28–42 weeks. A logistic regression model was used to estimate the associations of the measures of infant birth size and their Z-score with anemia under five years old. There were 26,802 (13.10%) children under five years old who were diagnosed has having anemia. Compared with children who did not have anemia, children who had anemia had a lower birth weight and smaller head circumference and a longer crown–heel length (all p-values < 0.05). After adjusting for confounders, not only birth weight (β coefficient, −0.008; 95% CI, −0.011–−0.004; p < 0.001) and head circumference (β coefficient, −0.004; 95% CI, −0.007–−0.001; p = 0.009), but also the related Z-scores were negatively associated with childhood anemia, while the trends for crown–heel length were the opposite. We further found significant interactions of folic acid use and maternal occupation with infant birth sizes. In conclusion, infants having abnormal sizes at birth are significantly associated with the risk for childhood anemia, which can be modified by folic acid use during pregnancy and maternal occupation.