During different stages of tumor development the immune system can either identify and destroy tumors, or promote their growth. Therapies targeting the immune system have emerged as a promising treatment modality for breast cancer, and immunotherapeutic strategies are being examined in preclinical and clinical models. However, our understanding of the complex interplay between cells of the immune system and breast cancer cells is incomplete. In this article, we review recent findings showing how the immune system plays dual host-protective and tumor-promoting roles in breast cancer initiation and progression. We then discuss estrogen receptor α (ERα)-dependent and ERα-independent mechanisms that shield breast cancers from immunosurveillance and enable breast cancer cells to evade immune cell induced apoptosis and produce an immunosuppressive tumor microenvironment. Finally, we discuss protumorigenic inflammation that is induced during tumor progression and therapy, and how inflammation promotes more aggressive phenotypes in ERα positive breast cancers.