To investigate immune escape during breast tumor progression, we analyzed the composition of leukocytes in normal breast tissues, ductal carcinoma in situ (DCIS), and invasive ductal carcinomas (IDC). We found significant tissue and tumor subtype-specific differences in multiple cell types including T cells and neutrophils. Gene expression profiling of CD45+CD3+ T cells demonstrated a decrease in CD8+ signatures in IDCs. Immunofluorescence analysis showed fewer activated GZMB+CD8+ T cells in IDC than in DCIS, including in matched DCIS recurrent IDC. TCR clonotype diversity was significantly higher in DCIS than in IDCs. Immune checkpoint protein TIGIT expressing T cells were more frequent in DCIS whereas high PD-L1 expression and amplification of CD274 (encoding PD-L1) was only detected in triple negative IDCs. Co-amplification of 17q12 chemokine cluster with ERBB2 subdivided HER2+ breast tumors into immunologically and clinically distinct subtypes. Our results show co-evolution of cancer cells and the immune microenvironment during tumor progression.