The prognostic impact of tumor-infiltrating lymphocytes (TILs) is intensively investigated in breast cancer (BC). It is already known that triple-negative breast cancer (TNBC), the most aggressive type of BC, has the highest percentage of TILs. In addition, there is an influence of steroid hormone receptor expression (type I nuclear receptors) on TIL subpopulations in breast cancer tissue. The link between type II nuclear receptors and the level of TILs is unclear. Therefore, the aim of this study was to quantify TILs in a panel of 264 sporadic breast cancers and investigate the correlation of TIL levels with type I and II nuclear receptors expression. TIL levels were significantly increased in the subgroup of TNBC. By contrast, they decreased in estrogen (ER)- or progesterone receptor (PR)-positive cases. Moreover, TIL levels were correlated with type II nuclear receptors, including PPARγ, with a significant inverse correlation of the nuclear form (r = −0.727, p < 0.001) and a weak positive correlation of the cytoplasmic form (r = 0.202, p < 0.002). Surprisingly, BC cases with a TIL Salgado score of >15% showed a significantly decreased overall survival. In addition, peritumoral inflammation was also quantified in BC tissue samples. In our cohort, although the level of peritumoral inflammation was not correlated with OS, it determined the prognostic value of ER, PR, and PPARγ in BC. Altogether, the present study provides a differentiated overview of the relations between nuclear receptor expression, TIL levels, peritumoral inflammation, and prognosis in BC.