The host’s immune system plays a pivotal role in many tumor types, including squamous cell carcinomas (SCCs). We aim to identify immunological prognosticators for lymph node metastases (LNM) and disease-specific survival (DSS) in penile SCC. For this retrospective observational cohort study, penile SCC patients (n = 213) treated in the Netherlands Cancer Institute, were selected if sufficient formalin-fixed, paraffin-embedded tumor material was available. Analysis included previously described high-risk human papilloma virus (hrHPV) status, immunohistochemical scores for classical and non-classical human leukocyte antigen (HLA) class I, programmed death ligand-1 (PD-L1) expression, and novel data on tumor-infiltrating macrophages and cytotoxic an regulatory T-cells. Clinicopathological characteristics and extended follow-up were also included. Regression analyses investigated relationships of the immune parameters with LNM and DSS. In the total cohort, diffuse PD-L1 tumor-cell expression, CD163+ macrophage infiltration, non-classical HLA class I upregulation, and low stromal CD8+ T-cell infiltration were all associated with LNM. In the multivariable model, only tumor PD-L1 expression remained a significant predictor for LNM (odds ratio (OR) 2.8, p = 0.05). hrHPV negativity and diffuse PD-L1 tumor-cell expression were significantly associated with poor DSS and remained so upon correction for clinical parameters [hazard ratio (HR) 9.7, p < 0.01 and HR 2.8, p = 0.03]. The only immune factor with different expression in HPV+ and HPV− tumors was PD-L1, with higher PD-L1 expression in the latter (p = 0.03). In the HPV− cohort (n = 158), LNM were associated with diffuse PD-L1 tumor-cell expression, high intratumoral CD163+ macrophage infiltration, and low number of stromal CD8+ T-cells. The first two parameters were also linked to DSS. In the multivariable regression model, diffuse PD-L1 expression remained significantly unfavorable for DSS (HR 5.0, p < 0.01). These results emphasize the complexity of the tumor microenvironment in penile cancer and point toward several possible immunotherapy targets. Here described immune factors can aid risk-stratification and should be evaluated in clinical immunotherapy studies to ultimately lead to patient tailored treatment.