Purpose: To investigate the diagnostic performance of endorectal ultrasound (ERUS), shear-wave elastography (SWE), and magnetic resonance imaging (MRI) in patients with a complex rectal adenoma or an early rectal cancer, i.e., T1 or T2 adenocarcinoma in a clinical setting, and to evaluate the association between SWE and stromal fraction (SF) and apparent diffusion coefficient (ADC) and SF. Method: This prospective study included patients undergoing ERUS and SWE for a rectal tumor subsequently confirmed by histopathology to be an adenoma or a T1 or T2 adenocarcinoma. The accuracy of the imaging methods was assessed by comparing the T category as determined by ERUS and MRI with histopathology, which served as the gold standard. SF was assessed on surgical specimens. Results: A total of 86 patients were included. Of these, 62 patients had adenomas and 24 patients had carcinomas, of which 11 were T1 tumors and 13 were T2 tumors. ERUS and MRI yielded sensitivity, specificity, and accuracy of 0.79 and 0.73, 0.95 and 0.90, and 0.86 and 0.78, respectively, for discrimination between benign and malignant lesions. The area under the receiver operating characteristics curve for SWE was 0.88, and with a cut-off value of 40 kPa the sensitivity, specificity, and accuracy were 0.79, 0.89, and 0.86, respectively. There was a positive correlation between SF and SWE with a p-value of <0.001 and a negative correlation between SF and ADC with a p-value of 0.011. Conclusion: Both ERUS and MRI classified T categories with a high accuracy; however, ERUS classified more adenomas correctly than MRI. In this small population, SWE could differentiate an adenoma from early carcinoma. SF was correlated with both SWE and ADC, as increasing SF tended to yield higher SWE and lower ADC values.