Immunotherapy success in colorectal cancer (CRC) is mainly limited to patients whose tumours exhibit high microsatellite instability (MSI). However, there is variability in treatment outcomes within this group, which is in part driven by the frequency and characteristics of tumour infiltrating immune cells. Indeed, the presence of specific infiltrating immune cell subsets has been shown to correlate with immunotherapy responses and is in many cases prognostic of treatment outcome. Tumour-infiltrating lymphocytes (TILs) can undergo distinct differentiation programs such as acquire features of tissue-residency or exhaustion, a process during which T cells upregulate inhibitory receptors such as PD-1 and loose functionality. While residency and exhaustion programs of CD8 T cells are relatively well-studied, these programs have only recently been appreciated in CD4 T cells and remain largely unknown in tumour-infiltrating natural killer (NK) cells. In this study, we use single cell RNA-seq data to identify signatures of residency and exhaustion in CRC infiltrating lymphocytes, including CD8, CD4 and NK cells. We then test these signatures in independent single cell data from tumour and normal tissue infiltrating immune cells. Further, we use versions of these signatures adapted for bulk RNA-seq data to identify a list of tumour intrinsic mutations associated with residency and exhaustion from TCGA data. Finally, using two independent transcriptomic data sets from patients with colon adenocarcinoma, we show that combinations of these signatures, in particular NK signatures, as well as tumour-associated signatures, such as TGF-β signalling, are associated with distinct survival outcomes in colorectal cancer patients.