Background: Identification of factors associated with quality of life (QOL) in breast cancer survivors can contribute to better functioning in this group of patients. The study aimed to assess the association between QOL and anthropometric, sociodemographic, and medical characteristics in postmastectomy women from southwestern Poland, 9.4 (±6.5) years after completed treatment. Materials and methods: QOL was estimated with the SF-36v2 questionnaire in 250 survivors aged 62.8 (±8.0) years with previously histologically confirmed invasive breast cancer. Results: Women in this study rated their overall QOL with an average of 60.7 (±17.9), Mental Component Summary of 62.8 (±19.8), and Physical Component Summary of 57.5 (±18.8) points. The use of multivariate regression analysis revealed that depression, chronic diseases, abdominal obesity, and pregnancy history have a statistically significant negative effect on the QOL of women after mastectomy, whereas participation in regular physical activity, living with a partner, the education level ≥12 years, and living in the city were associated with a higher QOL assessment. There were no significant relationships between QOL and the age, time since surgery, type of treatment, smoking, and occupational status of the patients. Conclusions: Health education, greater social support, specialist care in the treatment of comorbidities, and propagation of a physically active lifestyle can improve the physical and mental functioning of breast cancer survivors long after diagnosis and treatment.