Despite curative treatment and discharge from acute hospital settings, breast cancer patients often have cancer- and treatment-related morbidity which impairs them from returning to work. Hence, the role of community-based return to work rehabilitation programs is important to help these patients transition back to work. This was a retrospective cohort study involving patients with breast cancer conducted at a community-based cancer rehabilitation center. Patients were involved in an interdisciplinary vocational rehabilitation program involving physiatrists, occupational therapists, physiotherapists and social workers. We recruited 63 patients for this study cohort, with 46 (73.0%) patients ≤ 60 years old. After undergoing the rehabilitation program, there were 37 (58.7%) participants who successfully returned to work. These participants returned to work at either within 6 months (27.0%), 12 months (29.7%) or 24 months (43.2%) after enrollment into the program, with a majority enrolling in white collar jobs. Multivariate regression analysis revealed that significant negative factors for return to work were advanced stage of cancer (p = 0.004), along with clinically significant fatigue, measured on the Brief Fatigue Inventory (p < 0.001). However, perceived work ability (p = 0.020) was found to be a positive factor.