Cascade genetic testing of relatives from families with pathogenic variants associated with hereditary breast and ovarian cancer (HBOC) or Lynch syndrome (LS) has important implications for cancer prevention. We compared the characteristics of relatives from HBOC or LS families who did not have genetic testing (GT (−) group) with those who had genetic testing (GT (+) group), regardless of the outcome. Self-administered surveys collected cross-sectional data between September 2017 and December 2021 from relatives participating in the CASCADE cohort. We used multivariable logistic regression with LASSO variable selection. Among n = 115 relatives who completed the baseline survey, 38% (n = 44) were in the GT (−) group. Being male (OR: 2.79, 95% CI: 1.10–7.10) and without a previous cancer diagnosis (OR: 4.47, 95% CI: 1.03–19.42) increased the odds of being untested by almost three times. Individuals from families with fewer tested relatives had 29% higher odds of being untested (OR: 0.71, 95% CI: 0.55–0.92). Reasons for forgoing cascade testing were: lack of provider recommendation, lack of time and interest in testing, being afraid of discrimination, and high out-of-pocket costs. Multilevel interventions designed to increase awareness about clinical implications of HBOC and LS in males, referrals from non-specialists, and support for testing multiple family members could improve the uptake of cascade testing.