HCC, the most prevalent form of primary liver cancer, is prototypically an inflammation-driven cancer developing after years of inflammatory insults. Consequently, the hepatic microenvironment is a site of complex immunological activities. Moreover, the tolerogenic nature of the liver can act as a barrier to anti-tumor immunity, fostering cancer progression and resistance to immunotherapies based on immune checkpoint inhibitors (ICB). In addition to being a site of primary carcinogenesis, many cancer types have high tropism for the liver, and patients diagnosed with liver metastasis have a dismal prognosis. Therefore, understanding the immunological networks characterizing the tumor microenvironment (TME) of HCC will deepen our understanding of liver immunity, and it will underpin the dominant mechanisms controlling both spontaneous and therapy-induced anti-tumor immune responses. Herein, we discuss the contributions of the cellular and molecular components of the liver immune contexture during HCC onset and progression by underscoring how the balance between antagonistic immune responses can recast the properties of the TME and the response to ICB.