The traditional process of disease diagnosis from medical images follows a manual process, which is tedious and arduous. A computer‐aided diagnosis (CADs) system can work as an assistive tool to improve the diagnosis process. In this pursuit, this article introduces a unique architecture LPNet for classifying colon polyps from the colonoscopy video frames. Colon polyps are abnormal growth of cells in the colon wall. Over time, untreated colon polyps may cause colorectal cancer. Different convolutional neural networks (CNNs) based systems have been developed in recent years. However, CNN uses pooling to reduce the number of parameters and expand the receptive field. On the other hand, pooling results in data loss and is deleterious to subsequent processes. Pooling strategies based on discrete wavelet operations have been proposed in our architecture as a solution to this problem, with the promise of achieving a better trade‐off between receptive field size and computing efficiency. The overall performance of this model is superior to the others, according to experimental results on a colonoscopy dataset. LPNet with bio‐orthogonal wavelet achieved the highest performance with an accuracy of 93.55%. It outperforms the other state‐of‐the‐art (SOTA) CNN models for the polyps classification task, and it is lightweight in terms of the number of learnable parameters compared with them, making the model easily deployable in edge devices.