Deep learning applications in agriculture are advancing rapidly, leveraging data-driven learning models to enhance crop yield and nutrition. Tomato (Solanum lycopersicum), a vegetable crop, frequently suffers from pest damage and drought, leading to reduced yields and financial losses to farmers. Accurate detection and classification of tomato pests are the primary steps of integrated pest management practices, which are crucial for sustainable agriculture. This paper explores using Convolutional Neural Networks (CNNs) to classify tomato pest images automatically. Specifically, we investigate the impact of various optimizers on classification performance, including AdaDelta, AdaGrad, Adam, RMSprop, Stochastic Gradient Descent (SGD), and Nadam. A diverse dataset comprising 4263 images of eight common tomato pests was used to train and evaluate a customized CNN model. Extensive experiments were conducted to compare the performance of different optimizers in terms of classification accuracy, convergence speed, and robustness. RMSprop achieved the highest validation accuracy of 89.09%, a precision of 88%, recall of 85%, and F1 score of 86% among the optimizers, outperforming other optimizer-based CNN architectures. Additionally, conventional machine learning models such as logistic regression, random forest, naive Bayes classifier, support vector machine, decision tree classifier, and K-nearest neighbors (KNN) were applied to the tomato pest dataset. The best optimizer-based CNN architecture results were compared with these machine learning models. Furthermore, we evaluated the cross-validation results of various optimizers for tomato pest classification. The cross-validation results demonstrate that the Nadam optimizer with CNN outperformed the other optimizer-based approaches and achieved a mean accuracy of 79.12% and F1 score of 78.92%, which is 14.48% higher than the RMSprop optimizer-based approach. The state-of-the-art deep learning models such as LeNet, AlexNet, Xception, Inception, ResNet, and MobileNet were compared with the CNN-optimized approaches and validated the significance of our RMSprop and Nadam-optimized CNN approaches. Our findings provide insights into the effectiveness of each optimizer for tomato pest classification tasks, offering valuable guidance for practitioners and researchers in agricultural image analysis. This research contributes to advancing automated pest detection systems, ultimately aiding in early pest identification and proactive pest management strategies in tomato cultivation.