Tertiary lymphoid structures (TLSs), clusters of immune cells found around tumor tissue, have been shown to be associated with anti-tumor immunity, but the cellular composition within each TLS and whether the cellular composition of a TLS affects a patient's prognosis are poorly understood. In the present study, each TLS was categorized according to its cellular composition determined by a system of multiplex immunohistochemical staining and quantitative analysis, and the correlation between the category and prognosis was examined. Sixty-seven patients with curatively resected stage II/III colorectal cancer (CRC) were enrolled. A TLS, consisting of germinal center B cells, follicular dendritic cells, T helper (Th) cells, B cells, cytotoxic T cells, and macrophages, was confirmed in the tumor tissue of 58 patients (87%). The densities of Th cells and macrophages were significantly higher in relapsed patients than in not-relapsed patients (p = .043 and p = .0076). A higher ratio of Th cells was the most significant independent risk factor for disease relapse on multivariate analysis. The subset increasing in Th cells was GATA3 + Th2. A total of 353 TLSs was divided into five clusters according to immune cell composition. Among them, the Th-rich type TLS was significantly increased (p = .0009) in relapsed patients. These data suggest the possibility that Th cell-dominant composition might disturb the anti-tumor immune response, and the function of each TLS might differ depending on its composition.