Male infertility is a serious concern for public health, and the possible role of exposure to plasticizers such as phthalates and bisphenol A in contributing to the condition is widely debated. We have herein enrolled 155 infertility cases attending an infertility center and 211 controls (fathers of a spontaneously conceived newborn) to investigate this issue. The urinary levels of seven phthalates and BPA were analyzed through HPLC/MS/MS. All data were statistically elaborated considering information about clinical situation, life habits, occupational activity, and, for cases, semen parameters (volume, sperm concentration, total count of spermatozoa, and sperm motility). Results showed significantly higher urinary concentrations for all the phthalates in cases compared to controls, except for monoethylphthalate and BPA. In total, 90.07% of cases had sperm motility lower than the WHO reference value (2010), while 53.69%, 46.31%, and 16.56% had sperm total number, concentration, and volume, respectively, out of the reference range. Regarding the possible source of exposure, the use of scents seems to be a significant source of DEP (diethylphthalate). When considering occupational settings, industrial workers, dental technicians, artisans, and farmers using chemicals showed higher risk (OR = 2.766, 95% CI 1.236–6.185), particularly in relation to DnBP (di-n-butyl phthalate) and DEHP (di-ethyl-hexyl phthalate) exposure. No clear quantitative correlation between specific plasticizers and sperm parameters could be demonstrated but these findings call for future studies about the risks associated with exposure to their mixture.