Numerous studies showed that factors influencing fetal development and neonatal period could lead to lasting alterations in the brain of the offspring, in turn increasing the risk for eating disorders (EDs). This work aims to systematically and critically review the literature on the association of prenatal and perinatal factors with the onset of EDs in the offspring, updating previous findings and focusing on anorexia nervosa (AN) and bulimia nervosa (BN). A systematic literature search was performed on Pubmed, PsycINFO, and Scopus. The drafting of this systematic review was conducted following the PRISMA statement criteria and the methodological quality of each study was assessed by the MMAT 2018. A total of 37 studies were included in this review. The factors that showed a more robust association with AN were higher maternal age, preeclampsia and eclampsia, multiparity, hypoxic complications, prematurity, or being born preterm (< 32 weeks) and small for gestational age or lower birth size. BN was only associated with maternal stress during pregnancy. Many methodological flaws emerged in the considered studies, so further research is needed to clarify these inconsistencies. Altogether, data are suggestive of an association between prenatal and perinatal factors and the onset of EDs in the offspring. Nevertheless, given the methodological quality of the available literature, firm conclusions cannot be drawn and whether this vulnerability is specific to EDs or mental disorders remains to be defined. Also, a strong need for longitudinal and well-designed studies on this topic emerged.