(1) Background: The relatively poor expert restaging accuracy of MRI in rectal cancer after neoadjuvant chemoradiation may be due to the difficulties in visual assessment of residual tumor on post-treatment MRI. In order to capture underlying tissue alterations and morphologic changes in rectal structures occurring due to the treatment, we hypothesized that radiomics texture and shape descriptors of the rectal environment (e.g., wall, lumen) on post-chemoradiation T2-weighted (T2w) MRI may be associated with tumor regression after neoadjuvant chemoradiation therapy (nCRT). (2) Methods: A total of 94 rectal cancer patients were retrospectively identified from three collaborating institutions, for whom a 1.5 or 3T T2w MRI was available after nCRT and prior to surgical resection. The rectal wall and the lumen were annotated by an expert radiologist on all MRIs, based on which 191 texture descriptors and 198 shape descriptors were extracted for each patient. (3) Results: Top-ranked features associated with pathologic tumor-stage regression were identified via cross-validation on a discovery set (n = 52, 1 institution) and evaluated via discriminant analysis in hold-out validation (n = 42, 2 institutions). The best performing features for distinguishing low (ypT0-2) and high (ypT3–4) pathologic tumor stages after nCRT comprised directional gradient texture expression and morphologic shape differences in the entire rectal wall and lumen. Not only were these radiomic features found to be resilient to variations in magnetic field strength and expert segmentations, a quadratic discriminant model combining them yielded consistent performance across multiple institutions (hold-out AUC of 0.73). (4) Conclusions: Radiomic texture and shape descriptors of the rectal wall from post-treatment T2w MRIs may be associated with low and high pathologic tumor stage after neoadjuvant chemoradiation therapy and generalized across variations between scanners and institutions.