Background and Objectives: Colorectal cancer is the third most common oncological disease worldwide. The standard treatment of locally advanced rectal tumors is neoadjuvant radiochemotherapy in combination with surgical resection. The choice of specific treatment algorithm is highly dependent on MRI findings. The aim of this study is to show the potential role of ADC measurements in rectal cancer and their usage in different clinical scenarios. Materials and Methods: A total of 135 patients had rectal MRI evaluation. Seventy-five (56%) had histologically proven rectal adenocarcinoma and sixty (44%) were evaluated as rectal disease-free. An ADC measurement in the most prominent region of interest was obtained for all patients. Eighteen patients (24% of the rectal cancer group) had a second MRI after neoadjuvant chemoradiotherapy with comparison of the ADC values at the same region of interest as previously measured. Results: Rectal cancer ADC values were found to be significantly lower than the ones in the control group (p < 0.001). A statistically significant correlation was found when ADC values in rectal tumors of different T stages were compared (p = 0.039)—those with higher T stage as in locally advanced disease showed lower ADC values. Patients with extramural vascular invasion showed significantly lower ADC values (p = 0.01). There was a significant increase in ADC values after treatment (p < 0.001), and a negative correlation was observed (r = −0.6572; p = 0.004)—tumors with low initial ADC values showed a higher increase in ADC. Conclusions: ADC measurements have a complementary role in the assessment of rectal cancer and have the potential to predict the response to chemoradiotherapy and improve the planning of proper treatment strategies.