The etiology of prostate cancer, the second most common cancer in men globally, has a strong heritable component. While rare coding germline variants in several genes have been identified as risk factors from candidate gene and linkage studies, the exome-wide spectrum of causal rare variants remains to be fully explored. To more comprehensively address their contribution, we analysed data from 37,184 prostate cancer cases and 331,329 male controls from five cohorts with germline exome/genome sequencing and one cohort with imputed array data from a population enriched in low-frequency deleterious variants. Our gene-level collapsing analysis revealed that rare damaging variants in SAMHD1 as well as genes in the DNA damage response pathway (BRCA2, ATM and CHEK2) are associated with the risk of overall prostate cancer. We also found that rare damaging variants in AOX1 and BRCA2 were associated with increased severity of prostate cancer in a case-only analysis of aggressive versus non-aggressive prostate cancer. At the single variant level, we found rare non-synonymous variants in three genes (HOXB13, CHEK2, BIK) significantly associated with increased risk of overall prostate cancer and in four genes (ANO7, SPDL1, AR, TERT) with decreased risk. Altogether, this study provides deeper insights into the genetic architecture and biological basis of prostate cancer risk and severity.