Although immunotherapy has been demonstrated to be promising in triple-negative (TN) breast cancer (BC), most BC cases are classified as non-TN. To enrich the responders for immunotherapy regardless of their subtypes, classification based on tumor-infiltrating lymphocyte (TIL) levels and programmed death ligand-1 (PD-L1) status may be useful. However, this classification has not been fully applied to BC. Furthermore, suppressive subsets in the local tumor microenvironment, such as tumor-associated macrophages (TAMs), which promote tumor progression, cannot be ignored to overcome immunotherapy resistance. The aims of the present study were to classify primary BC cases based on the TIL levels and PD-L1 status, and to identify suppressive immune subsets in each categorized group. A retrospective analysis of 73 patients with invasive BC was performed. The frequency of TILs was evaluated in HE-stained slides (10% cutoff), and PD-L1 levels (SP142; 1% cutoff), as well as immune subsets (CD3
+
, CD8
+
, FOXP3
+
, CD20
+
, CD68
+
and CD204
+
cells) were assessed using immunohistochemistry. It was revealed that 22% (16/73) of the tumors were categorized as TIL
+
PD-L1
+
, of which 69% (11/16) were TN type. By contrast, 66% (48/73) of the tumors were categorized as TIL
−
PD-L1
−
, of which 77% (37/48) were HR
+
and HER2
−
types. The number of CD204
+
M2-type macrophages was significantly associated with high histological grade (P=0.0246) and high Ki-67 (P=0.0152), whereas CD68
+
macrophages were not associated with these factors. Furthermore, CD204
+
macrophages and FOXP3
+
Tregs accumulated in 88% (14/16) and 63% (10/16) of TIL
+
PD-L1
+
tumors, respectively, compared with 20.8% (10/48) and 27.1% (13/48) of TIL
−
PD-L1
−
tumors. In conclusion, 22% of BC tumors were classified as TIL
+
PD-L1
+
(69% were TN), which were enriched with suppressive immune subsets. These cell types may serve as potential novel immunotherapeutic targets.