A high-fat diet (HFD) and obesity are risk factors for many diseases including breast cancer. This is particularly important with close to 40% of the current adult population being overweight or obese. Previous studies have implicated that Mediterranean diets (MDs) partially protect against breast cancer. However, to date, the links between diet and breast cancer progression are not well defined. Therefore, to begin to define and assess this, we used an isocaloric control diet (CD) and two HFDs enriched with either olive oil (OOBD, high in oleate, and unsaturated fatty acid in MDs) or a milk fat-based diet (MFBD, high in palmitate and myristate, saturated fatty acids in Western diets) in a mammary polyomavirus middle T antigen mouse model (MMTV-PyMT) of breast cancer. Our data demonstrate that neither MFBD or OOBD altered the growth of primary tumors in the MMTV-PyMT mice. The examination of lung metastases revealed that OOBD mice exhibited fewer surface nodules and smaller metastases when compared to MFBD and CD mice. These data suggest that different fatty acids found in different sources of HFDs may alter breast cancer metastasis.