The driver genes regulating T-cell infiltration are important for understanding immune-escape mechanisms and developing more effective immunotherapy. However, researches in this field have rarely been reported in hepatocellular carcinoma (HCC). In the present study, we identified cancer driver genes triggered by copy number alterations such as CDKN2B, MYC, TSC1, TP53, and GSK3B. The T-cell infiltration levels were significantly decreased in both HCC and recurrent HCC tissues compared with the adjacent normal liver tissues. Remarkably, we identified that copy number losses of MAX and TP53 were candidate driver genes that significantly suppress T-cell infiltration in HCC. Accordingly, their downstream oncogenic pathway, cell cycle, was significantly activated in the low T-cell infiltration HCC. Moreover, the chemokine-related target genes by TP53, which played key roles in T-cell recruitment, were also downregulated in HCC with TP53/MAX deletions, suggesting that copy number losses in MAX and TP53 might result in T-cell depletion in HCC via downregulating chemokines. Clinically, the T-cell infiltration levels and chemokines activity could accurately predict the response of sorafenib, and the prognostic outcomes in HCC. In conclusion, the systematic analysis not only facilitates identification of driver genes and signaling pathways involved in T-cell infiltration and immune escape, but also gains more insights into the functional roles of T cells in HCC.