Background: Colorectal cancer (CRC) displays a complex pattern of inheritance. It is postulated that much of the missing heritability of CRC is enriched in high-impact rare alleles, which might play a crucial role in the etiology and susceptibility of CRC. Methods: In this study, an exome-wide association analysis was performed in 146 patients with high-risk CRC in the Middle East and 1395 healthy controls. The aim was to identify rare germline variants in coding regions and their splicing sites associated with high-risk CRC in the Middle Eastern population. Results: Rare inactivating variants (RIVs) in APC had the strongest association with high-risk CRC (6/146 in cases vs. 1/1395 in controls, OR = 59.7, p = 5.13 × 10−12), whereas RIVs in RIMS1, an RAS superfamily member, were significantly associated with high-risk CRC (5/146 case vs. 2/1395 controls, OR = 24.7, p = 2.03 × 10−8). Rare damaging variants in 17 genes were associated with high-risk CRC at the exome-wide threshold (p < 2.5 × 10−6). Based on the sequence kernel association test, nonsynonymous variants in six genes (TNXB, TAP2, GPSM3, ADGRG4, TMEM229A, and ANKRD33B) had a significant association with high-risk CRC. RIVs in APC—the most common high-penetrance genetic factor—were associated with patients with high-risk CRC in the Middle East. Individuals who inherited APC RIVs had an approximate 60-fold increased risk of developing CRC and were likely to develop the disease earlier. Conclusions: We identified new potential CRC predisposition variants in other genes that could play a role in CRC inheritance. However, large collaborative studies are needed to confirm the association of these variants with high-risk CRC. These results provide information for counseling patients with high-risk CRC and their families in our population.