Men of African descent have the highest prostate cancer (CaP) incidence and mortality rates, yet the genetic basis of CaP in African men has been understudied. We used genomic data from 3,963 CaP cases and 3,509 controls recruited in Ghana, Nigeria, Senegal, South Africa, and Uganda, to infer ancestry-specific genetic architectures and fine-mapped disease associations. Fifteen independent associations at 8q24.21, 6q22.1, and 11q13.3 reached genome-wide significance, including four novel associations. Intriguingly, multiple lead SNPs are private alleles, a pattern arising from recent mutations and the out-of-Africa bottleneck. These African-specific alleles contribute to haplotypes with odds ratios above 2.4. We found that the genetic architecture of CaP differs across Africa, with effect size differences contributing more to this heterogeneity than allele frequency differences. Population genetic analyses reveal that African CaP associations are largely governed by neutral evolution. Collectively, our findings emphasize the utility of conducting genetic studies that use diverse populations.