Curative radiotherapy for prostate cancer is common in the elderly. However, concerns about potential toxicity have inhibited access to radiotherapy for this population, for whom preserving quality of life (QoL) is crucial. The primary endpoint was to identify predictors of impaired QoL in men aged 75 years or older treated with curative intent radiotherapy with or without androgen deprivation therapy (ADT) for localized prostate cancer. We prospectively performed comprehensive geriatric assessment (CGA) and administered QoL questionnaires to 208 elderly (>75 years) patients prior to, plus two and six months after, radiotherapy (NCT 02876237). The median age of the patients was 77 years (range 75–89). At the start of the study, comorbidities were highlighted in 65% of patients: 23% were depressed, 23% had cognitive impairment, and 16% had reduced independence. At six months, 9% of patients had a consistently decreased QoL (>20 points), and a further 16% had a more moderate reduction (10 to 20 points) in QoL. None of the parameters studied (tumor characteristic, treatment, or oncogeriatric parameters) were predictive of a reduced QoL following radiotherapy. Though co-existing geriatric impairment was common, QoL was maintained for 75% of patients six months after radiotherapy. CGA was poorly predictive of tolerance of prostatic radiotherapy. Geriatric assessments dedicated to quality of life following radiotherapy need to be developed.