Introduction Childhood undernutrition is a major public health problem especially in low and middle-income countries (LMIC). The prevalence of early introduction of complementary feeding, low meal frequency, and low dietary diversity are frequent in LMICs. The effect of inappropriate complementary feeding practices on the nutritional status of children is not well documented in East African countries including Tanzania. Therefore, this study aimed at determining the effect of inappropriate complementary feeding practices on the nutritional status of children aged 6–24 months in urban Moshi, Tanzania. Methodology A retrospective cohort study was done using the Pasua and Majengo cohorts of mother-child pairs in urban Moshi who were enrolled from 2002 to 2017. About 3355 mother-child pairs were included in the analysis. Appropriate complementary feeding practices were assessed using WHO IYFP indicators such as age at introduction of solid, semi-solid, or soft foods, minimum dietary diversity, and minimum meal frequency. Nutritional status (stunting, wasting, and underweight) was determined. Multilevel modeling was applied to obtain the effect of inappropriate complementary feeding practices on the nutritional status of children and to account for the clustering effect of mothers and children and the correlation of repeated measures within each child. Results Majority of the children (91.2%) were given soft/semi-solid/solid foods before six months of age, 40.3percent had low meal frequency, and 74percent had low dietary diversity. Early introduction of complementary food at age 0–1 month was statistically significantly associated with higher risks of wasting and underweight (ARR 2.9, 95%CI 1.3–6.3; and ARR 2.6, 95% CI 1.3–5.1 respectively). Children with low minimum meal frequency had higher risks of stunting, wasting, and underweight (ARR 2.9, 95%CI 2.3–3.6; ARR 1.9, 95%CI 1.5–2.5 and ARR 1.9, 95%CI 1.5–2.4 respectively). Children with low minimum dietary diversity were more likely to be stunted than is the case with their peers who received the minimum dietary diversity (ARR 1.3, 95% CI 1.01–1.6). Conclusion There were a high proportion of children, which were fed inappropriately; Inappropriate complementary feeding practices predisposed children to undernutrition. Our study supports the introduction of complementary feeding, providing minimum dietary diversity, and minimum feeding frequency at six months of age as important in improving the nutritional status of the children.