The objective of this study is to investigate health-related quality of life (HRQoL) in male breast cancer patients. Data of 20,673 patients diagnosed with primary breast cancer (male: n = 84) who completed a questionnaire after discharge from hospital were analysed. HRQoL (SF-36), age, sex, education, native language, insurance status, and partnership status were measured. Cancer staging, treatment (partial mastectomy vs. radical mastectomy), and cancer site were indicated by the clinicians. The HRQoL scores of male breast cancer patients were compared with reference populations. Differences in HRQoL scores between men and women were compared using t tests and regression analysis. Compared to female breast cancer patients, male patients scored significantly higher on seven of eight subscales (physical functioning, role functioning-physical and emotional, bodily pain, vitality, social functioning, and mental health) in the regression analysis. Compared to the reference populations (general male population, men aged 61-70, and the cancer-affected population), male breast cancer patients scored lower on SF-36 subscales on average, with major differences in emotional and physical role functioning. The results suggest that male breast cancer patients may need early interventions that specifically target role functioning, which is severely impaired compared to the male reference population. Future research needs to assess HRQoL with cancer-specific questionnaires and longitudinal designs also focussing on male patients in breast centres.