There is limited evidence to support the relationship between the consumption of animal-source foods other than red meat and processed meat and colorectal cancer (CRC) risk. We aimed to examine the recent available evidence from observational studies about the association between these food groups’ intake and CRC risk. For this systematic review, we searched the PubMed database for the last five years. A total of fourteen cohort studies and seven case–control studies comprising a total of >60,000 cases were included. The studies showed a consistent significant decrease in CRC risk, overall and by subsites, associated with a high consumption of total dairy products. Less strong effects associated with the consumption of any subtype of dairy product were observed. Fish consumption, overall and by subtypes (oily or non-oily and fresh or canned), showed a mild inverse association with CRC risk. The association between white meat and egg intake and CRC risk was low and based on a small number of studies; thus, these findings should be interpreted with caution. In conclusion, a high consumption of total dairy products was associated with a lower CRC risk. However, evidence for fish, white meat, and eggs and the CRC risk were not as strong.