Little is known about changes in the abundance of tumor-infiltrating lymphocytes (TILs) and immune phenotype (IP) in recurrent and/or metastatic (R/M) head and neck squamous cell carcinoma (HNSCC). We aimed to compare the TILs and IP between initial and recurrent HNSCCs using paired analysis. Thirty-seven patients who experienced recurrence after surgical resection and received treatment with immune checkpoint inhibitors (ICIs) between June 2014 and June 2023 were included. Changes in intratumoral TIL (iTILs), stromal TIL (sTILs), and IPs were subjected to paired analysis between the initial and recurrent tumors. We investigated their relationship with the outcomes of ICIs. The density of iTIL and sTIL in the recurrent tumors was significantly lower compared to initial tumors. IP was significantly different; the proportion of desert IP was higher in recurrent tumors (83.8% vs. 35.1%, P < 0.001). Increased sTIL was a favorable indicator for overall response to ICIs and progression-free survival. Our findings suggest TILs decrease during recurrence compared with the initial tumor, resulting in a transition toward desert IP. Therefore, careful evaluation of TIL density in both initial and recurrent tumors is recommended when using ICIs in patients with R/M HNSCC.