To examine the impact of ultra-high iPSA levels of >50 ng/mL (uhPSA) after modern radiotherapy, we compared outcomes of 214 patients with uhPSA levels to 1161 other high-risk patients. Radiotherapy included brachytherapy ± external beam radiotherapy (EBRT) and EBRT alone (intensity-modulated radiotherapy or stereotactic body radiotherapy). The biochemical disease-free survival rate (bDFS), the distant metastasis-free survival rate (DMFS), local control, and pelvic lymph node control were analyzed. Patients with uhPSA levels had an inferior bDFS (84.8% at 5 years) and DMFS (93.9% at 5 years) compared to other high-risk patients (92.7% and 97.2%, both p < 0.001). The uhPSA group showed more distant metastases than the non-uhPSA group; however, the frequencies of local failure and pelvic lymph node recurrence were similar. The uhPSA group demonstrated hazard ratios (HRs) of 2.74 for bDFS and 2.71 for DMFS, similar to those of T3b–4 (HR 2.805 and 2.678 for bDFS and DMFS) and GS 9–10 (HR 2.280 and 2.743 for bDFS and DMFS). An uhPSA level could be a candidate for a single VHR factor to identify high-risk patients who require intensified treatment.