Aim Tumour budding (‘attacker’) and CD8+ T cells (‘defender’) are recognised as important parameters for risk stratification in colon cancers and, combined, may have an even stronger clinical impact. Here, we determine the value of tumour budding and CD8+ in rectal cancer patients treated with/without neoadjuvant therapy. Methods and results Using digital scans of all tumour slides/case, we analysed CD8+ T cell counts in two patient cohorts: 45 neoadjuvantly treated and 47 primarily surgically treated (totalling n = 543 slides) after double‐staining of the surgical resection specimen for pan‐cytokeratin and CD8+. Tumour buds in hot‐spots were manually counted (area = 0.785 mm2) and CD8+ T cell counts were analysed separately both in tumour budding hot‐spots and the densest CD8+ regions throughout the tumour. In neoadjuvantly treated patients, only tumour budding and not CD8+ T cells was associated with tumour features, including more advanced ypT (P = 0.0062), venous invasion (P = 0.002), lymphatic invasion (P = 0.0003) and perineural invasion (P = 0.0017), as well as higher American Joint Committee on Cancer (AJCC) tumour regression score (P = 0.0035), indicating less tumour response. Overall survival was also worse in patients with high‐grade budding in univariate analysis only. In contrast, all three variables, namely tumour budding (P = 0.0347), CD8+ T cells in budding hot‐spots (P = 0.0382) and CD8+ T cells in the densest areas (P = 0.0117) were also associated with worse (budding) and better (CD8) survival time in the multivariate setting. Conclusion In rectal cancer, tumour budding has clinical relevance in both primarily surgically treated patients and in those with neoadjuvantly treated patients, where it characterises highly aggressive residual disease. CD8+ T cell counts appear not to have prognostic relevance in the neoadjuvant context.