Purpose Cognitive dysfunction is reported in people with cancer. Therefore, we evaluated longitudinal changes in cognitive function and underlying mechanisms in people with colorectal cancer (CRC) and healthy controls (HCs). Patients and Methods Participants completed cognitive assessments and questionnaires reporting cognitive symptoms, fatigue, quality of life, and anxiety/depression at baseline (before chemotherapy, if given) and 6, 12, and 24 months. Blood tests included cytokines, clotting factors, apolipoprotein E genotype, and sex hormones. Primary end point was overall cognitive function measured by the Global Deficit Score at 12 months. Results We recruited 289 patients with localized CRC (173 received chemotherapy; median age, 59 years; 63% male), 73 patients with limited metastatic/recurrent CRC, and 72 HCs. Cognitive impairment was more frequent in patients with localized CRC than HCs at baseline (43% v 15%, respectively; P < .001) and 12 months (46% v 13%, respectively; P < .001), with no significant effect of chemotherapy. Attention/working memory, verbal learning/memory, and complex processing speed were most affected. Cognitive impairment was similar in patients with localized and metastatic CRC. Cytokine levels were elevated in patients with CRC compared with HCs. There was no association between overall cognitive function and fatigue, quality of life, anxiety/depression, or any blood test. Cognitive symptoms at 12 months were reported in 25% of patients with localized CRC versus 17% of HCs (P = .19). More participants who received chemotherapy had cognitive symptoms at 6 months (32%) versus those who did not (16%; P = .007), with no significant difference at 12 months (29% v 21%, respectively; P = .19). Objective cognitive function was only weakly associated with cognitive symptoms. Conclusion Patients with CRC had substantially more cognitive impairment at every assessment than HCs, with no significant added effect of chemotherapy. Mechanisms of cognitive impairment remain unknown.