Purpose. The incidence of early-onset rectal cancer (EORC) has been increasing since the past decade, while its underlying cause remained unknown. This study was aimed at clarifying the relationship between perirectal fat area (PFA) and EORC. Patients and Methods. All patients with rectal cancer who received radical excision between January 2016 and December 2017 at our hospital were included. The fat series images of pelvic magnetic resonance imaging scans were obtained and PFA at the ischial spine level was calculated using the ImageJ software. Results. A total of 303 patients were finally included and divided into two groups according to the median PFA: Group 1 (<20.2 cm2,
n
=
151
) and Group 2 (≥20.2 cm2,
n
=
152
). PFA positively correlated with body weight and body mass index. PFA increased with invasion depth, lymph node metastasis, TNM stage, tumor deposits, and vascular invasion. Patients with EORC had higher PFA than those with late-onset rectal cancer (LORC;
P
=
0.009
). Among patients with stage I–III rectal cancers, those in Group 2 had significantly shorter disease-free survival (
P
=
0.010
) and overall survival (
P
=
0.034
) than those in Group 1, and PFA was an independent predictor of disease-free survival (OR: 1.683 [1.126-3.015],
P
=
0.035
) and overall survival (OR: 1.678 [1.022-2.639],
P
=
0.046
). Conclusions. Patients with EORC had significantly higher PFA than those with LORC. PFA is positively correlated with T stage, N stage, TNM stage, tumor deposit, and vascular invasion and is an independent predictor of disease-free survival and overall survival. Therefore, perirectal fat may be involved in the carcinogenesis and development of EORC.