Objective: The objective was to examine the association between poor ovarian response to gonadotropin stimulation for in vitro fertilization (IVF) and adverse perinatal outcomes in singleton gestations in young patients. Methods: This was a retrospective cohort study including women aged 17–39 who underwent fresh embryo transfer and delivered a singleton neonate at a single center (pre-implantation genetic testing excluded) (2007–2022). Patients were classified as one of the following categories: poor responders—daily follicle-stimulating hormone (FSH) ≥ 150 IU yielding ≤ 3 retrieved oocytes; normal responders—4–15 oocytes; and high responders with ≥16 oocytes. The primary outcome was a composite of pre-eclampsia (mild or severe), small-for-gestational-age, gestational diabetes mellitus, and preterm birth (<37 weeks). We compared maternal and neonatal outcomes between the three groups. Multivariable logistic regression was used to control for confounders. Results: Overall, 507 women met the inclusion criteria. Of them, there were 44 (8.68%) poor responders, 342 (67.46%) normal responders, and 121 (23.87%) high responders. Poor responders, compared to normal and high responders, were characterized by a higher maternal age (34.64 ± 4.01 vs. 31.4 ± 5.04 vs. 30.01 ± 4.93, p < 0.001, respectively) and total FSH dosage (3028.41 ± 1792.05 IU vs. 2375.11 ± 1394.05 IU vs. 1869.31 ± 1089.63 IU, p < 0.001). The perinatal outcomes examined, including cesarean delivery (CD) rate and the composite outcome, were comparable between groups. Using multivariable logistic regression and adjusting for ovarian response group, maternal age, nulliparity, and estradiol level and endometrial thickness before ovulation triggering, poor response was not associated with CD rate or the composite outcome, with maternal age associated with CD (p = 0.005), and nulliparity with the composite outcome (p = 0.007). Similar results were obtained when comparing poor responders to each other group separately or to all other responders. Conclusions: Poor ovarian response is not associated with increased adverse maternal or neonatal outcomes.