Background: The predominant breast cancer subtypes, invasive lobular carcinoma (ILC) and invasive ductal carcinoma (IDC), have similar recurrence and survival rates but differing patterns of metastatic recurrence. Methods: A retrospective review of breast cancers treated at an academic medical center from 1999 to 2012 was performed. Demographic, pathologic, treatment, and follow-up data were collected for 179 ILC and 358 IDC patients (1:2 stage-matched). The median follow-up was 4.7 years. Results: The baseline characteristics were similar in the two groups. ILC was more likely to be hormone-receptor-positive/HER2-negative and mammographically occult. The number of surgical resections, breast conservation rate, systemic treatment, and taxane use was similar between the groups. The overall recurrence rate was the same. ILC recurred more often in the abdominal cavity (24.3% in ILC vs. 4.1% in IDC, p = 0.001). The disease-free survival and overall survival were equal. On multivariate analysis, age, stage of disease, hormone receptor status, and systemic therapy were associated with survival, but histology was not. Conclusions: Compared to ductal breast cancers, lobular breast cancers recur more often in the abdominal cavity. Both ILC and IDC have comparable surgical and medical treatment outcomes and survival. Our data suggest that enhanced surveillance and imaging might be useful in ILC.