Background: Colorectal cancer (CRC) patients experience multiple types of chemotoxicity affecting treatment compliance, survival, and quality of life (QOL). Prior research shows clinician-reported chemotoxicity (i.e., grading scales or diagnostic codes) predicts rehospitalization and cancer survival. However, a comprehensive synthesis of clinician-reported chemotoxicity is still lacking. Objectives: We conducted a systematic review and meta-analysis to determine chemotoxicity’s prevalence and risk factors in CRC. Methods: A systematic search from 2009 to 2024 yielded 30 studies for review, with 25 included in the meta-analysis. Results: Pooled prevalences of overall, non-hematological, and hematological moderate-to-severe toxicities were 45.7%, 39.2%, and 25.3%, respectively. The most common clinician-reported chemotoxicities were gastrointestinal (GI) toxicity (22.9%) and neuropathy or neutropenia (17.9%). Significant risk factors at baseline were malnutritional status, frailty, impaired immune or hepato-renal functions, short telomere lengths, low gut lactobacillus levels, age, female sex, aggressive chemotherapy, and low QOL. Age was associated with neutropenia (β: −1.44) and GI toxicity (β:1.85) (p-values < 0.01). Older adults (>65 y.o.) had higher prevalences of overall (OR: 1.14) and GI (OR: 1.65) toxicities, but a lower prevalence of neutropenia (OR: 0.65) than younger adults (p-values < 0.05). Conclusions. Our findings highlight the importance of closely monitoring and managing chemotoxicity in CRC patients receiving chemotherapy.