Purpose To study the incidence of tumor suppressor gene (TSG) mutations in men and women with impaired gametogenesis. Methods Gene association analyses were performed on blood samples in two distinct patient populations: males with idiopathic male infertility and females with unexplained diminished ovarian reserve (DOR). The male study group consisted of men with idiopathic azoospermia, oligozoospermia, asthenozoospermia, or teratozoospermia. Age-matched controls were men with normal semen analyses. The female study group consisted of women with unexplained DOR with anti-Müllerian hormone levels ≤ 1.1 ng/mL. Controls were age-matched women with normal ovarian reserve (> 1.1 ng/mL). Results Fifty-seven male cases (mean age = 38.4; mean sperm count = 15.7 ± 12.1; mean motility = 38.2 ± 24.7) and 37 agematched controls (mean age = 38.0; mean sperm count = 89.6 ± 37.5; mean motility = 56.2 ± 14.3) were compared. Variants observed in CHD5 were found to be enriched in the study group (p = 0.000107). The incidence of CHD5 mutation c.*3198_*3199insT in the 3′UTR (rs538186680) was significantly higher in cases compared to controls (p = 0.0255). 72 DOR cases (mean age = 38.7; mean AMH = 0.5 ± 0.3; mean FSH = 11.7 ± 12.5) and 48 age-matched controls (mean age = 37.6; mean AMH = 4.1 ± 3.0; mean FSH = 7.1 ± 2.2) were compared. Mutations in CHD5 (c.-140A>C), RB1 (c.1422-18delT, rs70651121), and TP53 (c.376-161A>G, rs75821853) were found at significantly higher frequencies in DOR cases compared to controls (p ≤ 0.05). In addition, 363 variants detected in the DOR patients were not present in the control group. Conclusion Unexplained impaired gametogenesis in both males and females may be associated with genetic variation in TSGs. TSGs, which play cardinal roles in cell-cycle control, might also be critical for normal spermatogenesis and oogenesis. If validated in larger prospective studies, it is possible that TSGs provide an etiological basis for some patients with impaired gametogenesis.