Hepatocellular carcinoma (HCC) is the major subtype of liver cancer, which ranks sixth in cancer incidence and third in mortality. Although great strides have been made in novel therapy for HCC, such as immunotherapy, the prognosis remains less than satisfactory. Increasing evidence demonstrates that the tumor immune microenvironment (TME) exerts a significant role in the evolution of HCC and has a non-negligible impact on the efficacy of HCC treatment. In the past two decades, the success in hematological malignancies made by chimeric antigen receptor-modified T (CAR-T) cell therapy leveraging it holds great promise for cancer treatment. However, in the face of a hostile TME in solid tumors like HCC, the efficacy of CAR-T cells will be greatly compromised. Here, we provide an overview of TME features in HCC, discuss recent advances and challenges of CAR-T immunotherapy in HCC.