A genetic diagnosis facilitates personalized cancer treatment and clinical care of relatives at risk, however, although 25% of colorectal cancer cases are familial, around 95% of the families are genetically unresolved. In this study, we performed gene panel analysis on germline DNA of 32 established or candidate colorectal cancer predisposing genes in 149 individuals from either families with an accumulation of colorectal cancers or families with only one sporadic case of very early onset colorectal cancer (≤40 years at diagnosis). We identified pathogenic or likely pathogenic genetic variants in 10.1% of the participants in genes such as APC, POLE, MSH2 or PMS2. The MSH2 variant, c.2168C>T, p.(Ser723Phe) was previously described as a variant of unknown significance, but we have now reclassified it to be likely pathogenic. The POLE variant, c.1089C>A, p.(Asn363Lys) was identified in a patient with three metachronous colorectal cancers from age 28 and turned out to be de novo. One pathogenic PMS2 variant was novel. We also identified a number of highly interesting variants of unknown significance in APC, BUB1, TP53 and RPS20. The RPS20 variant is novel and was found in a large Amsterdam I positive family with a multi tumor phenotype including 12 cases of CRC from as early as age 24. This variant was found to segregate with cancer in the family and multiple in silico tools predict it to be pathogenic. Our data further support the shift from phenotypic-based cancer panels to large panels including all established genes involved in hereditary cancer syndromes or (targeted) whole genome sequencing. Additionally, identification of a likely disease-predisposing variant in RPS20 expands the phenotypic spectrum of RPS20-related cancers and emphasize that this gene is relevant to include in colorectal cancer gene panels.