The indications for stereotactic body radiotherapy (SBRT) for prostate cancer have increased. However, the relationships between adverse events and risk factors remain unclear. This study aimed to clarify associations between adverse events and dose index for prostate SBRT. Participants comprised 145 patients irradiated with 32–36 Gy in 4 fractions. Radiotherapy-related risk factors such as dose-volume histogram parameters and patient-related risk factors such as T stage and Gleason score were evaluated in a competing risk analysis. Median follow-up duration was 42.9 months. A total of 9.7% had acute Grade ≥ 2 GU toxicities and 4.8% had acute Grade ≥ 2 GI toxicities. A total of 11.1% had late Grade ≥ 2 GU toxicities and 7.6% had late Grade ≥ 2 GI toxicities. Two (1.4%) patients suffered from late Grade 3 GU toxicities. Similarly, two (1.4%) patients suffered from late Grade 3 GI toxicities. Acute GU and GI events correlated with prostate volume and dose to the hottest 10 cc volume (D10cc)/volumes receiving a minimum of 30 Gy (V30 Gy) of rectum, respectively. Late GI toxicity, frequency, and rectal hemorrhage correlated with rectal D0.1 cc/D1 cc, maximum dose to the bladder, and rectal D0.1 cc, respectively. Toxicities after prostate SBRT using 32–36 Gy/4 fractions were acceptable. Our analysis showed that acute toxicities correlated with volume receiving a medium dose level, and late toxicities correlated with highest point dose of organs at risk.