Despite thorough characterizations of cellular compositions within the breast tumor microenvironment (TME), their implications for disease progression and patient prognosis are still poorly understood. Unraveling these effects is vital for identifying potential targets to improve treatment outcomes. In this study, we devised an explainable machine learning (XML) pipeline to scrutinize the associations between TME cellular constituents and relapse-free survival (RFS). By applying our pipeline to estimated cell fractions in the METABRIC and TCGA datasets and comparing these results with associations to pathological complete response (pCR) after neoadjuvant chemotherapy (NAC), we created a comprehensive catalog of the TME role based on 5000 patient samples. Our findings reveal an unexpected dichotomy in which macrophages correlate positively with pCR but negatively with RFS, particularly within estrogen receptor-positive (ER+) and Luminal A and B (LumA/B) cancer subtypes. We show that this pattern is driven by heterogeneity in breast tumors characterized by increasing levels of macrophage infiltration. Through imaging mass cytometry (IMC) analysis, we discovered that macrophages tend to accumulate in the vicinity of HLA-ABChiepithelial cells as their frequency increases in tumor tissues and also express elevated levels of HLA-ABC protein. Combining IMC with single-cell RNA sequencing (scRNA-seq) data, we uncovered a significant association between these HLA-ABChimacrophages and regulatory and exhausted T cells (TRegand TEx), suggesting their involvement in immune suppression, likely by creating a chronically activated immunosuppressive TME. Subsequent cell-cell communication analysis predicted interactions between HLA-ABChimacrophages and TExcells via the ligands SIGLEC9, ALCAM, and CSF1, and with TRegcells through APP, ANGPTL4, and SIGLEC9 signaling. Considering the clinical relevance of macrophages in ER+ (LumA/B) subtypes, our research enhances the characterization of macrophage-driven immune suppression in these tumors and identifies potential targets for immunomodulatory strategies.