Background: We retrospectively analyzed the 5-year biochemical disease-free survival (bDFS) and occurrence of late toxicity in prostate cancer patients treated with pencil beam scanning (PBS) proton radiotherapy. Methodology: In the period from January 2013 to June 2018, 853 patients with prostate cancer were treated with an ultra-hypofractionated schedule (36.25 GyE/five fractions). The mean PSA value was 6.7 (0.7–19.7) µg/L. There were 318 (37.3%), 314 (36.8%), and 221 (25.9%) patients at low (LR), favorable intermediate (F-IR), and unfavorable intermediate risk (U-IR), respectively. Neoadjuvant hormonal therapy was administered to 197 (23.1%) patients, and 7 (0.8%) patients had adjuvant hormonal therapy. The whole group of patients reached median follow-up time at 62.7 months, and their mean age was 64.8 (40.0–85.7) years. The bDFS rates and late toxicity profile were evaluated. Results: Median treatment time was 10 (7–38) days. Estimated 5-year bDFS rates were 96.5%, 93.7%, and 91.2% for low-, favorable intermediate-, and unfavorable intermediate-risk groups, respectively. Cumulative late toxicity (CTCAE v4.0) of G2+ was as follows: gastrointestinal (GI)—G2: 9.1%; G3: 0.5%; genitourinary (GU)—G2: 4.3%, and no G3 toxicity was observed. PSA relapse was observed in 58 (6.8%) patients: 16 local, 22 lymph node, 4 bone recurrences, and 10 combined sites of relapse were detected. Throughout the follow-up period, 40 patients (4.7%) died, though none due to prostate cancer. Conclusion: Ultra-hypofractionated proton beam radiotherapy is an effective treatment for low- and favorable intermediate-risk prostate cancer, with long-term bDFS rates comparable to other techniques. It is promising for unfavorable intermediate-risk prostate cancer and has acceptable long-term GI and favorable GU toxicity.