Background To investigate the prognosis of females with invasive ductal carcinoma (IDC), invasive lobular carcinoma (ILC), and mixed invasive ductal and lobular carcinoma (IDLC) according to hormone receptor (HR) and HER2 status. Methods Data of 171,881 patients from the SEER database were analyzed. Propensity score matching was used to balance the covariates. Breast cancer-specific survival (BCSS) and overall survival (OS) of IDC, ILC, and IDLC were investigated. Results Patients with ILC were older, had lower tumor grade, higher tumor stage, larger tumor size, more nodal metastasis, higher estrogen receptor(+), lower HER2(−), and less likely to receive partial mastectomy and chemotherapy compared with IDC and IDLC. ILC and IDLC showed better prognosis than IDC after matching by Kaplan-Meier curves. Multivariate Cox regression showed better OS of ILC and IDLC compared with IDC with hazard ratio and a 95% confidence interval of 0.84 (0.77-0.90) and 0.91 (0.83-1.00), respectively. For HR(+)HER2(−) subgroup, ILC showed better OS than IDC; IDC showed worse BCSS and OS than IDLC. For HR(+)HER2(+); ILC showed better OS compared with IDLC; there were no survival differences of IDC, ILC, and IDLC for HER2(+). For HR(−)HER2(−), ILC and IDC showed better BCSS and OS compared with IDLC by multivariate analysis. Conclusions The prognoses of female patients with IDC, ILC or IDLC were associated with the molecular subtypes of breast carcinoma. Management decisions should be based on pathological types and molecular subtypes.