This study aimed at assessing the correctness of a caregiver’s perception of their child’s diet status and to determine the factors which may influence their judgment. 815 child-caregiver pairs were recruited from two primary schools. 3-day 24-h recall was used to evaluate children’s dietary intake, Chinese Children Dietary Index (CCDI) was used to evaluate the dietary quality. Multivariate logistic regression models were used to explore the factors that could influence the correctness of caregiver’s perception. In the current study, 371 (62.1%) children with “high diet quality” and 35 (16.1%) children with “poor diet quality” were correctly perceived by their caregivers. Children who were correctly perceived as having “poor diet quality” consumed less fruits and more snacks and beverages than those who were not correctly perceived (p < 0.05). Obese children were more likely to be correctly identified as having “poor diet quality” (OR = 3.532, p = 0.040), and less likely to be perceived as having “high diet quality”, even when they had a balanced diet (OR = 0.318, p = 0.020). Caregivers with a high level of education were more likely to correctly perceive children’s diet quality (OR = 3.532, p = 0.042). Caregivers in this study were shown to lack the ability to correctly identify their children’s diet quality, especially amongst children with a “poor diet quality”. Obesity, significantly low consumption of fruits or high consumption of snacks can raise caregivers’ awareness of “poor diet quality”.