Neoadjuvant chemo-radiotherapy (CRT) followed by total mesorectal excision (TME) represents the standard treatment for patients with locally advanced (≥ T3 or N+) rectal cancer (LARC). Approximately 15% of patients with LARC shows a complete response after CRT. The use of pre-treatment MRI as predictive biomarker could help to increase the chance of organ preservation by tailoring the neoadjuvant treatment. We present a novel machine learning model combining pre-treatment MRI-based clinical and radiomic features for the early prediction of treatment response in LARC patients. MRI scans (3.0 T, T2-weighted) of 72 patients with LARC were included. Two readers independently segmented each tumor. Radiomic features were extracted from both the “tumor core” (TC) and the “tumor border” (TB). Partial least square (PLS) regression was used as the multivariate, machine learning, algorithm of choice and leave-one-out nested cross-validation was used to optimize hyperparameters of the PLS. The MRI-Based “clinical-radiomic” machine learning model properly predicted the treatment response (AUC = 0.793, p = 5.6 × 10–5). Importantly, the prediction improved when combining MRI-based clinical features and radiomic features, the latter extracted from both TC and TB. Prospective validation studies in randomized clinical trials are warranted to better define the role of radiomics in the development of rectal cancer precision medicine.