The recent demonstration of modest single-agent activity of programmed death-ligand 1 (PD-L1) and programmed death receptor-1 (PD-1) antibodies in patients with breast cancer has generated hope that breast cancer can be made amenable to immunotherapy. Depending on the subtype of breast cancer, it is now clear in both primary and metastatic disease that the extent of tumor-infiltrating T cells is not only prognostic for survival but predictive of response to non-immune, standard therapies. Despite these findings, immune cytolytic activity in spontaneous breast tumors, the burden of nonsynonymous tumor mutations, and the predicted load of neo-epitopes – factors linked to response to checkpoint blockade in other malignancies – are all relatively modest in breast cancer, compared to melanoma or lung cancer. Thus, in breast cancer, combinations of immune agents with non-redundant mechanisms of action are the high-priority strategies. For most breast cancers that exhibit relatively modest T cell infiltration, major challenges include immune suppression in the tumor microenvironment as well as failed or suboptimal T cell priming. Agents that trigger de novo T cell responses may be critical for successful development of cancer immunotherapy and immune prevention in breast cancer. Success may also require reaching beyond nonsynonymous mutations as the T cell epitopes to target, especially as numerous unmutated proteins were validated as breast cancer associated antigens in the pre-checkpoint era. A deeper understanding of the immunobiology of breast cancer will be critical for immunotherapy to become broadly relevant in this disease.