The aim of this study was to evaluate the association of adipose tissue characteristics with survival in rectal cancer patients. All consecutive patients, diagnosed with stage II–IV rectal cancer between 2010–2016 using baseline unenhanced Computed Tomography (CT), were included. Baseline total, subcutaneous and visceral adipose tissue areas (TAT, SAT, VAT) and densities (TATd, SATd, VATd) at third lumbar vertebra (L3) were retrospectively measured. The association of these tissues with cancer-specific and progression-free survival (CCS, PFS) was assessed by using competitive risk models adjusted by age, sex and stage. Among the 274 included patients (median age 70 years, 41.2% females), the protective effect of increasing adipose tissue area on survival could be due to random fluctuations (e.g., sub-distribution hazard ratio—SHR for one cm2 increase in SAT = 0.997; 95%confidence interval—CI = 0.994–1.000; p = 0.057, for CSS), while increasing density was associated with poorer survival (e.g., SHR for one Hounsfield Unit—HU increase in SATd = 1.03, 95% CI = 1.01–1.05, p = 0.002, for CSS). In models considering each adipose tissue area and respective density, the association with CSS tended to disappear for areas, while it did not change for TATd and SATd. No association was found with PFS. In conclusion, adipose tissue density influenced survival in rectal cancer patients, raising awareness on a routinely measurable variable that requires more research efforts.