The tumor microenvironment (TME), including immune cells, cancer-associated fibroblasts, endothelial cells, adjacent normal cells, and others, plays a crucial role in influencing tumor behavior and progression. Here, we characterized the TME in 83 primary renal tumors and matched metastatic or recurrence tissue samples (n = 15) from papillary renal cell carcinoma (pRCC) types 1 (n = 20) and 2 (n = 49), collecting duct carcinomas (CDC; n = 14), and high-grade urothelial carcinomas (HGUC; n = 5). We investigated 10 different markers of immune infiltration, vasculature, cell proliferation, and epithelial-to-mesenchymal transition by using machine learning image analysis in conjunction with immunohistochemistry. Marker expression was compared by Mann-Whitney and Kruskal-Wallis tests and correlations across markers using Spearman's rank correlation coefficient. Multivariable Poisson regression analysis was used to compare marker expression between histological types, while accounting for variation in tissue size. Several immune markers showed different rates of expression across histological types of renal carcinoma. Using pRCC1 as reference, the incidence rate ratio (IRR) of CD3+ T cells (IRR [95% confidence interval, CI] = 2.48 [1.53-4.01]) and CD20+ B cells (IRR [95% CI] = 4.38 [1.22-5.58]) was statistically significantly higher in CDC. In contrast, CD68+ macrophages predominated in pRCC1 (IRR [95% CI] = 2. 35 [1.42-3.9]). Spatial analysis revealed CD3+ T-cell and CD20+ B-cell expressions in CDC to be higher at the proximal (p < 0.0001) and distal (p < 0.0001) tumor periphery than within the central tumor core. In contrast, expression of CD68+ macrophages in pRCC2 was higher in the tumor center compared to the proximal (p = 0.0451) tumor periphery and pRCC1 showed a distance-dependent reduction, from the central tumor, in CD68+ macrophages with the lowest expression of CD68 marker at the distal tumor periphery (p = 0.004). This study provides novel insights into the TME of rare kidney cancer types, which are often understudied. Our findings of differences in marker expression and localization by histological subtype could have implications for tumor progression and response to immunotherapies or other targeted therapies.