The impact of longitudinal anthropometric changes during adjuvant chemotherapy on long-term survival in non-metastatic colon cancer is unclear. Herein, we analyzed the prognostic significance of computed tomography (CT)-measured body composition changes in colon cancer patients who underwent surgery followed by adjuvant FOLFOX (folinic acid, 5-fluorouracil, oxaliplatin) chemotherapy. Data of 167 patients with stage III or high-risk stage II colon cancer were analyzed. Skeletal muscle index (SMI), skeletal muscle radiodensity (SMR), visceral fat index (VFI), subcutaneous fat index (SFI), and total fat index (TFI) changes during chemotherapy were calculated using preoperative and postchemotherapy CT image data. The Cox proportional hazard model was used to determine the correlation between changes in anthropometric values and overall survival (OS). The median changes (%) in SMI, SMR, VFI, SFI, and TFI over 210 days during chemotherapy were 8.7% (p < 0.001), 3.4% (p = 0.001), −19% (p < 0.001), −3.4% (p = 0.936), and −11.9% (p < 0.001), respectively. Cut-off values of changes in SMI (skeletal muscle index change, SMIC) and SMR (skeletal muscle radiodensity change, SMRC) were defined at −2% and −2 Hounsfield units (HU) respectively, whereas those of changes in VFI (visceral fat index change, VFIC), SFI (subcutaneous fat index change, SFIC), and TFI (total fat index change, TFIC) were based on values that provided the largest χ2 on the Mantel–Cox test. Multivariable analysis revealed that low SMR measured on a postchemotherapy CT scan (hazard ratio, HR: 0.32, 95% confidence interval, CI: 0.15–0.70, p = 0.004) and visceral fat loss of at least 46.57% (HR: 0.31, 95% CI: 0.14–0.69, p = 0.004) were independent poor prognostic factors for OS. Severe visceral fat loss during FOLFOX chemotherapy and low skeletal muscle radiodensity measured on postchemotherapy CT scans are associated with poor OS in stage III and high-risk stage II colon cancer patients.