While detailed analysis of aberrant cancer cell signaling pathways and changes in cancer cell DNA has dominated the field of breast cancer biology for years, there now exists increasing evidence that the tumor microenvironment (TME) including tumor-infiltrating immune cells support the growth and development of breast cancer and further facilitate invasion and metastasis formation as well as sensitivity to drug therapy. Furthermore, breast cancer cells have developed different strategies to escape surveillance from the adaptive and innate immune system. These include loss of expression of immunostimulatory molecules, gain of expression of immunoinhibitory molecules such as PD-L1 and HLA-G, and altered expression of components involved in apoptosis. Furthermore, the composition of the TME plays a key role in breast cancer development and treatment response. In this review we will focus on i) the different immune evasion mechanisms used by breast cancer cells, ii) the role of immune cell infiltration in this disease, and (iii) implication for breast cancer-based immunotherapies.