Tumor tissue is composed of tumor cells and tumor stroma. Tumor stroma contains various immune cells and non-immune stromal cells, forming a complex tumor microenvironment which plays pivotal roles in regulating tumor growth. Recent successes in immunotherapies against tumors, including immune checkpoint inhibitors, have further raised interests in the immune microenvironment of liver carcinoma. The immune microenvironment of tumors is formed because of interactions among tumor cells, immune cells and non-immune stromal cells, including fibroblasts and endothelial cells. Different patterns of immune microenvironment are observed among different tumor subtypes, and their clinicopathological significance and intertumor/intratumor heterogeneity are being intensively studied. Here, we review the immune microenvironment of hepatocellular carcinoma, intrahepatic cholangiocarcinoma and liver metastasis of colorectal adenocarcinoma, focusing on its histopathological appearance, clinicopathological significance, and relationship with histological and molecular classifications. Understanding the comprehensive histopathological picture of a tumor immune microenvironment, in addition to molecular and genetic approaches, will further potentiate the effort for precision medicine in the era of tumor-targeting immunotherapy.