Pain related to cancer or treatment is a critical quality of life (QOL) issue for breast cancer survivors. In a prospective study of 375 breast cancer patients (enrolled during 2008–2014), we characterized the risk factors for adjuvant radiotherapy (RT)-associated pain. Pain score was assessed at pre- and post-RT as the mean of four pain severity items (i.e., pain at its worst, least, average, and now) from the Brief Pain Inventory (BPI) with 11-point numeric rating scale (0–10). Pain scores of 4–10 were considered clinically-relevant pain. The study consists of 58 non-Hispanic whites (NHW; 15%), 78 black or African Americans (AA; 21%), and 239 Hispanic whites (HW; 64%). Overall, the prevalence of clinically-relevant pain was 16% at pre-RT, 31% at post-RT, and 20% RT-associated increase. In univariate analysis, AA and HW had significantly higher pre- and post-RT pain compared to NHW. In multivariable logistic regression analysis, pre-RT pain was significantly associated with HW and obesity; post-RT pain was significantly associated with AA, HW, younger age, ≥2 comorbid conditions, above median hotspot volume receiving >105% prescribed dose, and pre-RT pain score ≥4. RT-associated pain was significantly associated with AA (odds ratio [OR]=3.27; 95% confidence interval (CI)=1.09–9.82), younger age (OR=2.44, 95% CI=1.24–4.79), and 2 or ≥3 comorbid conditions (OR=3.06, 95%CI=1.32–7.08; OR=4.61, 95%CI=1.49–14.25, respectively). These risk factors may help to guide RT decision making process, such as hypo-fractionated RT schedule. Furthermore, effective pain management strategies are needed to improve QOL in breast cancer patients with clinically-relevant pain.