Improved risk stratification is needed for patients with localized prostate cancer. This study characterized and assessed the prognostic potential of distinct immune cell infiltration patterns in the prostate tumor microenvironment. Using tissue microarrays, multiplex immunohistochemistry/immunofluorescence, and automated digital pathology, we analyzed radical prostatectomy specimens from two large patient cohorts (training: n = 470; validation: n = 333) to determine infiltration levels of seven immune cell types in malignant versus benign prostate tissue: CD3+CD8−FoxP3− T helper cells, CD3+CD8+FoxP3− cytotoxic T cells (CTLs), CD3+CD8−FoxP3+ regulatory T cells (Tregs), CD20+ B cells, CD68+CD163− M1 macrophages, CD68+CD163+ M2 macrophages, and tryptase+ mast cells. Results were further validated by cell type enrichment analyses of bulk tumor RNAseq data from a third independent patient cohort (n = 99). Prognostic potential was assessed by Kaplan–Meier and uni‐/multi‐variate Cox regression analyses. Clinical endpoint was biochemical recurrence. All seven immune cell types were enriched in prostate cancer versus benign stroma, while there was selective enrichment for B cells, Tregs, M1 and M2 macrophages, and depletion of mast cells and CTLs in prostate cancer epithelium. In all three cohorts, high levels of infiltrating Tregs, M1, and M2 macrophages in stroma and/or epithelium were associated with biochemical recurrence (p < 0.05; log‐rank test). After adjustment for routine clinical variables, Tregs and M2 macrophages remained significant adverse predictors of biochemical recurrence (p < 0.05; multivariate Cox regression). Our comprehensive analyses of immune cell infiltration patterns in the prostate tumor microenvironment highlight infiltrating Tregs, M1, and M2 macrophages as adverse predictors of prostate cancer outcome. © 2021 The Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.