Poor nutrition during the first two years of life has long term consequences, but resource-poor households often do not have the means to access nutrient-dense and diverse diets. Pooled data of 24-h dietary recalls (n = 3336) and 2019 retail food prices were analyzed to determine associations of retail cost and energy cost (per 100 kcal) with diet quality indicators for diets of 6–24-month-old South African children who were breastfed (BF-diet) and not breastfed (NBF-diet) during the 24-h recall period. Compared to the BF-diet, retail cost for the NBF-diet was three times higher for age 6–11 months, and double for age 12–17 months. Higher retail cost and energy cost were both associated with higher mean adequacy ratios and dietary diversity scores for BF and NBF diets, except energy cost for the NBF-diet for age 6–11 months. Overall, inclusion of flesh foods, vitamin A-rich fruits and vegetables, and other fruit and vegetables increased from the lowest to the highest retail cost tertile. The higher cost of more nutritious diets highlights the importance of the affordability of diets in interventions aimed at improving diet quality. Possible strategies include: identifying the most-affordable foods within each food group, focusing on foods that provide multiple key micronutrients, and the inclusion of locally available indigenous foods.