Aside from surgical resection, locally advanced rectal cancer is regularly treated with neoadjuvant chemoradiotherapy. Since the concept of cancer treatment has shifted from only focusing on tumor cells as drivers of disease progression towards a broader understanding including the dynamic tumor microenvironment (TME), the impact of radiotherapy on the TME and specifically the tumor immune microenvironment (TIME) is increasingly recognized. Both promoting as well as suppressing effects on anti-tumor immunity have been reported in response to rectal cancer (chemo-)radiotherapy and various targets for combination therapies are under investigation. A literature review was conducted searching the PubMed database for evidence regarding the pleiotropic effects of (chemo-)radiotherapy on the rectal cancer TIME, including alterations in cytokine levels, immune cell populations and activity as well as changes in immune checkpoint proteins. Radiotherapy can induce immune-stimulating and -suppressive alterations, potentially mediating radioresistance. The response is influenced by treatment modalities, including the dosage administered and the highly individual intrinsic pre-treatment immune status. Directly addressing the main immune cells of the TME, this review aims to highlight therapeutical implications since efficient rectal cancer treatment relies on personalized strategies combining conventional therapies with immune-modulating approaches, such as immune checkpoint inhibitors.