The immunosuppressive tumor microenvironment (TME) consists of suppressive cells producing a variety of immunomodulatory proteins, such as programmed death ligand 1 (PD-L1) and indoleamine-2,3-dioxygenase (IDO). Although granzyme B (GrB) is known to convey the cytolytic activities of CD8+ cytotoxic lymphocytes, it is also expressed by other cells, such as regulatory T and B cells, for immunosuppressive purposes. The role of GrB+ lymphocytes in melanoma has not been examined extensively. In this study, benign, premalignant, and malignant melanocytic tumors were stained immunohistochemically for CD8 and GrB. PD-L1 was also stained from malignant samples that had accompanying clinicopathological data. The association of CD8+ and GrB+ lymphocytes with PD-L1 expression, tumor stage, prognosis, and previously analyzed immunosuppressive factors were evaluated. Our aim was to obtain a more comprehensive perception of the immunosuppressive TME in melanoma. The results show that both CD8+ and GrB+ lymphocytes were more abundant in pT4 compared to pT1 melanomas, and in lymph node metastases compared to primary melanomas. Surprisingly, a low GrB/CD8 ratio was associated with better recurrence-free survival in primary melanomas, which indicates that GrB+ lymphocytes might represent activated immunosuppressive lymphocytes rather than cytotoxic T cells. In the present study, CD8+ lymphocytes associated positively with both tumor and stromal immune cell PD-L1 and IDO expression. In addition, PD-L1+ tumor and stromal immune cells associated positively with IDO+ stromal immune and melanoma cells. The data suggest that IDO and PD-L1 seem to be key immunosuppressive factors in CD8+ lymphocyte-predominant tumors in CM.