One of the most common cancers worldwide, primary liver cancer remains a major cause of cancer-related mortality. Hepatocellular carcinoma and cholangiocarcinoma represent the majority of primary liver cancer cases. Despite advances in the development of novel anti-cancer therapies that exploit targets within the immune system, survival rates from liver cancer remain poor. Furthermore, responses to immunotherapies, such as immune checkpoint inhibitors, have revealed limited and variable responses amongst patients with hepatocellular carcinoma, although combination immunotherapies have shown recent breakthroughs in clinical trials. This has shifted the focus towards improving our understanding of the underlying immune and molecular characteristics of liver tumours that may influence their response to immune-modulating treatments. In this review, we outline the complex interactions that occur in the tumour microenvironment of hepatocellular carcinoma and cholangiocarcinoma, respectively, from a histopathological perspective. We explore the potential role of a classification system based on immune-specific characteristics within each cancer type, the importance of understanding inter- and intra-tumoural heterogeneity and consider the future role of histopathology and novel technologies within this field.