Insect pests are a major global factor affecting agricultural crop productivity and quality. Rapid and precise insect pest detection is crucial for improving handling and prediction techniques. There are several methods for pest detection and classification tasks; still, the inaccurate detection, computation complexity and several other challenges affect the performance of the model. Thus, this research presents a Deep Learning (DL) approach that has led to significant advancements and is currently being applied successfully in many domains, such as autonomous insect pest detection. Initially, the input images are gathered from the test dataset. The next step in pre‐processing the input images is to improve the model capacity by removing unwanted data using the Enhanced Kuan filter method. Then, the pre‐processed images are segmented using the Attention‐based U‐Net method. Finally, a novel Attention Based Reptile Residual Capsule Auto Encoder (ARRCAE) technique is proposed to classify and recognize crop pests. Furthermore, the Improved Reptile Search Optimisation (IRSO) algorithm is employed to fine‐tune the classification parameters optimally. As a result, the proposed study enhances performance by classifying crop pest detection systems. The suggested method makes use of a Python tool for simulation, and pest datasets are utilized for result analysis. The suggested model beats other current models with an accuracy of 98%, precision of 97%, recall of 96%, and specificity of 99% for the pest dataset, per the simulation results that were obtained.This article is protected by copyright. All rights reserved.