(1) Objectives: The primary objective is to compare the rate of large-for-gestational-age (LGA) between women with diet-controlled gestational diabetes mellitus (GDM) and those with non-GDM, and to assess whether or not diet-controlled GDM is an independent factor of LGA fetuses. The secondary objectives are to compare the rates of other common adverse pregnancy outcomes, such as preeclampsia, cesarean section rate, preterm birth, and low Apgar score, between pregnancies with diet-controlled GDM and non-GDM pregnancies. (2) Methods: A retrospective cohort study was conducted on singleton pregnancies, diagnosed with GDM and non-GDM between 24 and 28 weeks of gestation, based on a two-step screening test. The prospective database of the obstetric department was accessed to retrieve the records meeting the inclusion criteria, and full medical records were comprehensively reviewed. The patients were categorized into two groups, GDM (study group) and non-GDM (control group). The main outcome was the rate of LGA newborns, and the secondary outcomes included pregnancy-induced hypertension, preterm birth, cesarean rate, low Apgar scores, etc. (3) Results: Of 1364 recruited women, 1342 met the inclusion criteria, including 1177 cases in the non-GDM group and 165 (12.3%) in the GDM group. Maternal age and pre-pregnancy BMI were significantly higher in the GDM group. The rates of LGA newborns, PIH, and cesarean section were significantly higher in the GDM group (15.1% vs. 7.1%, p-value < 0.001; 7.8% vs. 2.6%, p-value = 0.004; and 54.5% vs. 41.5%, p-value = 0.002; respectively). On logistic regression analysis, GDM was not significantly associated with LGA (odds ratio 1.64, 95% CI: 0.97–2.77), while BMI and gender were still significantly associated with LGA. Likewise, GDM was not significantly associated with the rate of PIH (odds ratio: 1.7, 95% CI: 0.825–3.504), while BMI and maternal age were significantly associated with PIH, after controlling confounding factors. (4) Conclusions: The rates of LGA newborns, PIH, and cesarean section are significantly higher in women with diet-controlled GDM than those with non-GDM. Nevertheless, the rates of LGA newborns and PIH are not directly caused by GDM but mainly caused high pre-pregnancy BMI and advanced maternal age, which are more commonly encountered among women with GDM.