Exposure to endocrine-disrupting chemicals (EDCs) may result in oxidative stress and endocrine system disturbance, which can have an impact on human reproduction and development. In male reproductive health, EDCs have been related to impaired reproductive function and male infertility, altered fetal development, and testicular germ-cell, prostate, and breast cancers. We conducted an electronic search using PubMed on endocrine disruptors related to oxidative stress and male infertility, and evaluated their association with endocrine-disrupting chemicals (bisphenols, phthalates, and parabens) in 25 articles. Higher levels of urinary bisphenols showed correlation with impaired semen quality and increased DNA damage. Considering phthalates and their metabolites, all studies found a positive association between urinary levels of phthalates and at least one semen parameter indicative of low semen quality; some studies also revealed sperm DNA damage. The studies on parabens less often revealed correlation of urinary parabens concentrations with a decrease in sperm count, as well as motility and DNA damage. Moreover, EDCs can elevate ROS production and lipid peroxidation, increase apoptosis, induce epigenetic modifications, and change the Y:X sperm chromosome ratio and sperm protein composition. Our review revealed detrimental effects of EDCs on semen quality and sperm DNA integrity—especially in BPA and phthalates, but also in parabens.