Male breast cancer (MBC) patients seem to have inferior survival compared to female (FBC) ones, which is not fully explained by usual prognostic factors. Recurrence analysis could show differences in relapse patterns and/or in patients' approaches that justify these outcomes. Retrospective analysis of MBC patients treated in a cancer center between 1990 and 2014, looking for relapse. For each patient, three matched FBC patients were selected by: diagnosis' year, age (within 5 years), stage and tumors' type (only luminal-like were considered). Differences between cohorts were assessed by χ(2) test and hierarchical clustering was performed to define subgroups according to relapse local. Survival curves were calculated by Kaplan-Meier and compared using log-rank test. Statistical significance was defined as p < 0.05. Groups were balanced according to age, histological grade, stage, expression of hormonal receptors and adjuvant treatments. Median time to recurrence was equivalent, p = 0.72, with the majority of patients presented with distant metastases, p = 0.69, with more lung involvement in male, p = 0.003. Male patients were more often proposed to symptomatic treatment (21.1% vs. 4.4%, p = 0.02). Overall and from recurrence survivals were poorer for male, median: 5 years [95% confidence interval (CI): 4.1-5.9 years] and 1 year (95% CI: 0-2.1 years) vs. 10 years (95% CI: 7.8-12.2 years) and 2 years (95% CI: 1.6-2.4 years), p < 0.001 and p = 0.004, respectively, and this tendency remained in the five cluster subgroups, that identified five patterns of relapse, p = 0.003. MBC patients had the worst survival, even after controlling important factors, namely the local of relapse. Palliative systemic treatment had favorable impact in prognosis and its frequently avoidance in male could justify the outcomes differences.