Background: Both genetic and dietary factors play significant roles in the etiology of colorectal cancer (CRC). To evaluate the relationship between certain food exposures and the risk of CRC, we carried out a large-scale association analysis in the UK Biobank. Methods: The associations of 139 foods and nutrients’ intake with CRC risk were assessed among 118,210 participants. A polygenic risk score (PRS) of CRC was created to explore any interaction between dietary factors and genetic susceptibility in CRC risk. The hazard ratio (HR) and 95% confidence interval (CI) of CRC risk linked to dietary variables and PRS were estimated using Cox regression models. Multiple comparisons were corrected using the error discovery rate (FDR). Results: During a mean follow-up of 12.8 years, 1466 incidents of CRC were identified. In the UK Biobank, alcohol and white bread were associated with increased CRC risk, and their HRs were 1.08 (95% CI: 1.03–1.14; FDRP = 0.028) and 1.10 (95% CI: 1.05–1.16; FDRP = 0.003), whereas dietary fiber, calcium, magnesium, phosphorus, and manganese intakes were inversely associated. We found no evidence of any PRS–nutrient interaction relationship in relation to CRC risk. Conclusions: Our results show that higher intakes of alcohol and white bread are associated with increased CRC risk, whilst dietary fiber, calcium, magnesium, phosphorus, and manganese are inversely associated.