Purpose Assessment of health-related quality of life (HRQOL) is a major concern among breast cancer survivors. There is little knowledge about HRQO and its determinants among Saudi women with breast cancer (BC). This study aimed to describe HRQOL in Saudi women with breast cancer and explore the association of HRQO with sociodemographic and clinical variables. Methods A cross-sectional study involving 147 BC participants who fulfilled a 36-item short form health survey (SF-36) administered by the Medical Outcomes Study. An analysis of multivariable linear logistic regression was conducted to determine what factors are associated with HRQOL. Results For the physical component scale, general health scored the highest (55.41 ± 17.99) while role physical function scored the lowest (48.61 ± 22.51). The mean scores in the mental component were higher for social function (56.91 ± 20.84), while the lowest score was observed for energy and vitality (49.62 ± 20.84). Significant differences were observed in many SF-36 domains across sociodemographic and clinical variables. Older age, having had a mastectomy as well as comorbid conditions, and receiving chemotherapy and hormonal therapy were associated with decreased HRQOL. However, being physically active, employed, and having more education were associated with higher HRQOL. Conclusion HRQOL is generally poor among breast cancer survivors in Saudi Arabia, depending on a variety of factors. These findings highlight the need for routine QOL assessment. Furthermore, recognizing these predictors may play a key role in maximizing HRQOL for breast cancer participants.