For breast cancer survivors return to work (RTW) is important from an economic, societal and personal perspective. Thus, we investigated the impact of side effects and other factors on RTW. Five years post‐diagnosis 135 disease‐free breast cancer survivors below retirement age who were employed pre‐diagnosis recorded their current and previous working status and reasons for impaired RTW. Patient‐reported outcomes were prospectively reported over the cancer continuum. One year post‐surgery 57% of survivors worked the same and 22% with reduced working time compared to pre‐diagnosis. Logistic regression revealed significant associations of depressive symptoms, arm morbidity, lower education and younger age with impaired RTW after 1 year, and persisting physical fatigue and living with partner with impaired RTW after 5 years. Major self‐reported reasons included fatigue and cognitive problems. Temporal patterns of general quality of life (QoL), physical, cognitive and role function, and financial problems were significantly worse among women with no RTW compared to those working again. In conclusion, cessation of work after breast cancer seems associated with worse QoL. Fatigue, psychological and cognitive problems as well as arm morbidity seemed to hinder RTW. Thus, a better management of these problems might help women to stay in working life.