Background: High variability in the age at cancer diagnosis in Lynch syndrome (LS) patients is widely observed, even among relatives with the same germline pathogenic variant (PV) in the mismatch repair (MMR) genes. Genetic polymorphisms and lifestyle factors are thought to contribute to this variability. We investigated the influence of previously reported genetic polymorphisms on the age at cancer diagnosis in a homogenous LS cohort with a South African founder germline PV c.1528C>T in the MLH1 gene. Methods: A total of 359 LS variant heterozygotes (LSVH) from 60 different families were genotyped for specific genetic polymorphisms in GSTM1, GSTT1, CYP1A1, CYP17, PPP2R2B, KIF20A, TGFB1, XRCC5, TNF, BCL2, CHFR, CDC25C, ATM, TTC28, CDC25C, HFE, and hTERT genes using Multiplex Polymerase Chain Reaction and MassArray methods. Kaplan–Meier survival analysis, univariate and multivariate Cox proportional hazards gamma shared frailty models adjusted for sex were used to estimate the association between age at cancer diagnosis and polymorphism genotypes. A p-value < 0.05 after correcting for multiple testing using the Benjamini–Hochberg method was considered significant at a 95% confidence interval. Results: We identified three genotypes in the cell-cycle regulation, DNA repair, and xenobiotic-metabolism genes significantly associated with age at cancer diagnosis in this cohort. The CYP1A1 rs4646903 risk (GG) and CDC25C rs3734166 polymorphic (GA+AA) genotypes were significantly associated with an increased risk of a younger age at cancer diagnosis (Adj HR: 2.03 [1.01–4.08], p = 0.034 and Adj HR: 1.53 [1.09–2.14], p = 0.015, respectively). LSVH who were heterozygous for the XRCC5 rs1051685 SNP showed significant protection against younger age at cancer diagnosis (Adj HR: 0.69 [CI, 0.48–0.99], p = 0.043). The risk of a younger age at any cancer diagnosis was significantly high in LS carriers of one to two risk genotypes (Adj HR: 1.49 [CI: 1.06–2.09], corrected p = 0.030), while having one to two protective genotypes significantly reduced the risk of developing any cancer and CRC at a younger age (Adj HR: 0.52 [CI: 0.37–0.73], and Adj HR: 0.51 [CI: 0.36–0.74], both corrected p < 0.001). Conclusions: Polymorphism genotypes in the cell-cycle regulation, DNA repair, and xenobiotic metabolizing genes may influence the age at cancer diagnosis in a homogenous LS cohort with a South African founder germline PV c.1528C>T in the MLH1 gene.