Even low levels of toxic metal exposure (As, Cd, Hg, and Pb) in infancy might be harmful to children’s development. This study investigated toxic metal exposure on healthy weaning-age infants and its relationship with growth, diet, and iron/anemia status. The weight, height, head circumference, whole blood levels of four toxic metals, hemoglobin, and serum ferritin of healthy infants was measured. Among 210 infants with a median age of 11.4 months (interquartile range: 10.5–12.0), the median levels of As, Cd, Hg, and Pb were 1.2 μg/L, 0.05 μg/L, 0.8 μg/L, and 0.83 μg/dL, respectively. In adjusted linear regression models, post-birth weight gain (Pb) and current head circumference (As, Pb) were negatively associated with toxic metal levels. In multiple linear regression or logistic regression analysis, the duration of breastfeeding (all four metals), perceived adequacy of rice-based food intake (As), regular fish intake (As, Hg), and iron deficiency with/without anemia (Cd, Pb) were associated with increased toxic metal levels. Although levels of toxic metals may not usually be high in this population, individual exposure risk may need to be assessed after considering the type of feeding or intake of complementary foods and the iron/anemia status while evaluating growth status during late infancy.