The ALFRED (Allelic Loss Featuring Rare Damaging) in silico method was developed to identify cancer predisposition genes through the identification of somatic second hits. By applying ALFRED to ~10,000 tumor exomes, 49 candidate genes were identified. We aimed to assess the causal association of the identified genes with colorectal cancer (CRC) predisposition. Of the 49 genes, NSD1, HDAC10, KRT24, ACACA and TP63 were selected based on specific criteria relevant for hereditary CRC genes. Gene sequencing was performed in 736 patients with familial/early onset CRC or polyposis without germline pathogenic variants in known genes. Twelve (predicted) damaging variants in 18 patients were identified. A gene-based burden test in 1596 familial/early-onset CRC patients, 271 polyposis patients, 543 TCGA CRC patients and >134,000 controls (gnomAD, non-cancer), revealed no clear association with CRC for any of the studied genes. Nevertheless, (non-significant) over-representation of disruptive variants in NSD1, KRT24 and ACACA in CRC patients compared to controls was observed. A somatic second hit was identified in one of 20 tumors tested, corresponding to an NSD1 carrier. In conclusion, most genes identified through the ALFRED in silico method were not relevant for CRC predisposition, although a possible association was detected for NSD1, KRT24 and ACACA.