The tumor microenvironment (TME) and its multifaceted interactions with cancer cells are major targets for cancer treatment. Single-cell technologies have brought major insights into the TME, but the resulting complexity frequently precludes conclusions on function. Therefore, we combined single-cell RNA sequencing and spatial transcriptomic data to explore the relationship between different cancer-associated fibroblast (CAF) populations and immune cell exclusion in breast tumors. Our data show for the first time the degree of spatial organization of different CAF populations in breast cancer. We found that IL-iCAFs, Detox-iCAFs, and IFN-beta-iCAFs tended to cluster together, while Wound-myCAFs, TGF-beta-myCAFs, and ECM-myCAFs formed another group that overlapped with elevated TGF-beta signaling. Differential gene expression analysis of areas with CD8+ T-cell infiltration/exclusion within the TGF-beta signaling-rich zones identified elastin microfibrillar interface protein 1 (EMILIN1) as a top modulated gene. EMILIN1, a TGF-beta inhibitor, was upregulated in IFN-beta-iCAFs directly modulating TGF-beta immunosuppressive function. Histological analysis of 74 breast cancer samples confirmed that high EMILIN-1 expression in the tumor margins was related to high CD8+ T-cell infiltration, consistent with our spatial gene expression analysis. High EMILIN-1 expression was also associated with better prognosis of patients with breast cancer, underscoring its functional significance for the recruitment of cytotoxic T cells into the tumor area. In conclusion, our data show that correlating TGF-beta signaling to a CAF subpopulation is not enough because proteins with TGF-beta-modulating activity originating from other CAF subpopulations can alter its activity. Therefore, therapeutic targeting should remain focused on biological processes rather than on specific CAF subtypes.