Tumor-specific T cells likely underpin effective immune checkpoint-blockade therapies. Yet, most studies focus on Treg cells and CD8+ tumor-infiltrating lymphocytes (TILs). Here we study CD4+ TILs in human lung and colorectal cancers and observe that non-Treg CD4+ TILs average more than 70% of total CD4+ TILs in both cancer types. Leveraging high dimensional analyses including mass cytometry and single-cell sequencing, we reveal that CD4+ TILs are heterogeneous at both gene and protein levels, within each tumor and across patients. Consistently, we find different subsets of CD4+ TILs showing characteristics of effectors, tissue resident memory (Trm) or exhausted cells (expressing PD-1, CTLA-4 and CD39). In both cancer types, the frequencies of CD39− non-Treg CD4+ TILs strongly correlate with frequencies of CD39− CD8+ TILs, which we and others have previously shown to be enriched for cells specific for cancer-unrelated antigens (bystanders). Ex-vivo, we demonstrate that CD39− CD4+ TILs can be specific for cancer unrelated antigens, such as HCMV epitopes. Overall, our findings highlight that CD4+ TILs cells are not necessarily tumor-specific and suggest measuring CD39 expression as a straightforward way to quantify or isolate bystander CD4+ T cells.Graphical abstract