We performed a meta analysis to access the relationship of estrogen receptor of alpha (ESRα) polymorphisms with the risk of prostate cancer (PC). Twenty-four case-control studies (including 5477 cases and 10708 controls) were recruited for meta-analysis. The strongest association with the risk of PC was observed between ESRα rs9340799 and rs2234693 under the two genotypic models of allele and codominance in the overall population (p < 0.05). Under the subgroup analysis of ethnicity, we observed that ESRα rs9340799 was significantly associated with the susceptibility to PC in European population (AvsG, p = 0.000; AAvsGG, p = 0.002), while there was no difference in Asian (AvsG, p = 0.493; AAvsGG, p = 0.736) or African population (AvsG, p = 0.800; AAvsGG, p = 0.788). The results also showed that significant association between rs2234693 and the susceptibility to PC in European (CvsT, p = 0.004; CCvsTT, p = 0.001) and Asian population (CvsT, p = 0.004; CCvsTT, p = 0.003), but not in African population (CvsT, p = 0.636; CCvsTT, p = 0.669). The meta-analysis indicated that ESRα rs9340799 and rs2234693 might contribute to susceptibility and development of PC in European population.