In this work, we propose a computer-aided diagnosis system for automatic classification of adenomatous polyps from colonic histological images. The proposed method employs an ensemble of ConvNeXt variants, which is one of the most recent and prominent convolution based deep learning architectures. Comprehensive experiments show that classification performance of the proposed method outperforms the state-of-the-art Deep CNN models on our dataset, with a 95% accuracy. It achieves 91.1% and 90% classification accuracy respectively on EBHI and UniToPatho datasets, which shows that the model can generalize well for other datasets as well. Furthermore, we investigate the effect of stain normalization preprocessing on the classification accuracy. While the other methods are sensitive to the normalization, the proposed method retains its accuracy levels under the variations, which also indicates a high generalization ability. Lastly, we use the Grad-Cam method and show that the proposed model gives more attention to the regions where the cancer indicators potentially reside.