According to the World Health Organization, lung and colon cancers are known for their high mortality rates which necessitate the diagnosis of these cancers at an early stage. However, the limited availability of data such as histopathology images used for diagnosis of these cancers, poses a significant challenge while developing computer‐aided detection system. This makes it necessary to keep a check on the number of parameters in the artificial intelligence (AI) model used for the detection of these cancers considering the limited availability of the data. In this work, a customised compact and efficient convolution transformer architecture, termed, C3‐Transformer has been proposed for the diagnosis of colon and lung cancers using histopathological images. The proposed C3‐Transformer relies on convolutional tokenisation and sequence pooling approach to keep a check on the number of parameters and to combine the advantage of convolution neural network with the advantages of transformer model. The novelty of the proposed method lies in efficient classification of colon and lung cancers using the proposed C3‐Transformer architecture. The performance of the proposed method has been evaluated on the ‘LC25000’ dataset. Experimental results shows that the proposed method has been able to achieve average classification accuracy, precision and recall value of 99.30%, 0.9941 and 0.9950, in classifying the five different classes of colon and lung cancer with only 0.0316 million parameters. Thus, the present computer‐aided detection system developed using proposed C3‐Transformer can efficiently detect the colon and lung cancers using histopathology images with high detection accuracy.