The clinical impact of tumor-infiltrating lymphocytes (TILs) is less known for breast cancer patients with the estrogen receptor-positive (ER+)/human epidermal growth factor receptor-negative (HER−) subtype. Here, we explored the prognostic and predictive value of TILs regarding distant recurrence-free interval (DRFI) and breast cancer-specific survival (BCSS) in 763 postmenopausal patients randomized to receive tamoxifen vs. no systemic treatment. TILs were assessed in whole section tumor samples stained with H&E and divided into low (<10%), intermediate (10–39%), or high (≥40%). High TILs were associated with poor prognostic variables and good prognoses for all patients, but not within the ER+/HER2− group. Within the ER+/HER2− group, high gene expression of CD19 and PD-L1 and high IMMUNE1 score indicated good prognosis in multivariable analysis while high CD8 and CD19 gene expression and high IMMUNE1 score were associated with less tamoxifen benefit. These results indicate that within the ER+/HER2− subtype there could be subsets of patients where expression of specific TIL markers might be used to reveal candidates for immune therapy interventions upon failure of the endocrine therapy.