Provision of social support and rehabilitation for patients with physical, mental and functional problems after cancer treatment is important for long-term health-related quality of life (HRQOL). Effective use of human and financial healthcare resources requires identification of patients requiring rehabilitation. The objectives of the current study were to clarify the patterns of physical and psychosocial recovery over time, to evaluate the associations among baseline variables, treatment-related factors and HRQOL at 6 months, 1 and 2 years after breast cancer surgery, and to identify the significant factors predicting HRQOL at each point. A multicenter longitudinal study was performed to evaluate physical conditions, anxiety, depression and HRQOL at one month (baseline), 6 months, and 1 and 2 years after surgery in 196 patients (mean age: 53.3 years old) with early breast cancer and no postoperative recurrence. Physical conditions were evaluated using a patient-reported symptom checklist. HRQOL was rated using the Functional Assessment of Cancer Treatment scale-General (FACT-G) and the Breast Cancer subscale (FACT-B). Anxiety and depression were rated using the Hospital Anxiety and Depression Scale (HADS).More than 50% of patients had local problems of "tightness", "arm weakness" and "arm lymphedema", and systemic problems of "reduced energy, fatigue, and general weakness" postoperatively. The HRQOL score significantly improved one year after surgery, and scores for physical, emotional and functional well-being also increased with time, whereas the score for social well-being was highest at baseline and decreased with time. Depression and anxiety significantly improved with time. Concomitant disease, marital status and the presence of a partner, anxiety and depression at baseline, pathological lymph node involvement, and adjuvant intravenous chemotherapy were significant factors predicting FACT-G scores at 6 months and 1 and 2 years after surgery. Depression at baseline was a strong predictor of HRQOL up to 2 years after surgery. These results suggest that physical rehabilitation is required for tightness and lymphedema to improve long-term postoperative physical function. A further study of psychosocial interventions is required to improve depression and social well-being after breast cancer surgery.