Sugar, particularly as free sugars or sugar-sweetened beverages, significantly contributes to total energy intake, and, possibly, to increased body weight. Excessive consumption may be considered as a proxy of poor diet quality. However, no previous studies evaluated the association between the habit of adding sugars to “healthy” foods, such as plain milk and fresh fruit, and indicators of adiposity and/or dietary quality in children. To answer to these research questions, we Panalysed the European cohort of children participating in the IDEFICS study. Anthropometric variables, frequency of consumption of sugars added to milk and fruit (SAMF), and scores of adherence to healthy dietary pattern (HDAS) were assessed at baseline in 9829 children stratified according to age and sex. From this cohort, 6929 children were investigated again after two years follow-up. At baseline, a direct association between SAMF categories and adiposity indexes was observed only in children aged 6–<10 years, while the lower frequency of SAMF consumption was significantly associated with a higher HDAS. At the two year follow-up, children with higher baseline SAMF consumption showed significantly higher increases in all the anthropometric variables measured, with the exception of girls 6–<10 years old. The inverse association between SAMF categories and HDAS was still present at the two years follow-up in all age and sex groups. Our results suggest that the habit to adding sugars to foods that are commonly perceived as healthy may impact the adherence to healthy dietary guidelines and increase in adiposity risk as well.