A growing body of research has found close links between the human gut microbiota and colorectal cancer (CRC), associated with the direct actions of specific bacteria and the activities of microbiota-derived metabolites, which are implicated in complex immune responses, thus influencing carcinogenesis. Diet has a significant impact on the structure of the microbiota and also undergoes microbial metabolism. Some metabolites, such as short-chain fatty acids (SCFAs) and indole derivatives, act as protectors against cancer by regulating immune responses, while others may promote cancer. However, the specific influence of these metabolites on the host is conditional. We reviewed the recent insights on the relationships among diet, microbiota-derived metabolites, and CRC, focusing on their intricate immunomodulatory responses, which might influence the progression of colorectal cancer.