Purpose To evaluate the association between a previously published 313 variant–based breast cancer (BC) polygenic risk score (PRS313) and contralateral breast cancer (CBC) risk, in BRCA1 and BRCA2 pathogenic variant heterozygotes. Methods We included women of European ancestry with a prevalent first primary invasive BC (BRCA1 = 6,591 with 1,402 prevalent CBC cases; BRCA2 = 4,208 with 647 prevalent CBC cases) from the Consortium of Investigators of Modifiers of BRCA1/2 (CIMBA), a large international retrospective series. Cox regression analysis was performed to assess the association between overall and ER-specific PRS313 and CBC risk. Results For BRCA1 heterozygotes the estrogen receptor (ER)-negative PRS313 showed the largest association with CBC risk, hazard ratio (HR) per SD = 1.12, 95% confidence interval (CI) (1.06–1.18), C-index = 0.53; for BRCA2 heterozygotes, this was the ER-positive PRS313, HR = 1.15, 95% CI (1.07–1.25), C-index = 0.57. Adjusting for family history, age at diagnosis, treatment, or pathological characteristics for the first BC did not change association effect sizes. For women developing first BC < age 40 years, the cumulative PRS313 5th and 95th percentile 10-year CBC risks were 22% and 32% for BRCA1 and 13% and 23% for BRCA2 heterozygotes, respectively. Conclusion The PRS313 can be used to refine individual CBC risks for BRCA1/2 heterozygotes of European ancestry, however the PRS313 needs to be considered in the context of a multifactorial risk model to evaluate whether it might influence clinical decision-making.