Castor (Ricinus communis L.) is an important nonedible industrial crop that produces oil, which is used in the production of medicines, lubricants, and other products. However, the quality and quantity of castor oil are critical factors that can be degraded by various insect pest attacks. The traditional method of identifying the correct category of pests required a significant amount of time and expertise. To solve this issue, automatic insect pest detection methods combined with precision agriculture can help farmers in providing adequate support for sustainable agriculture development. For accurate predictions, the recognition system requires a sufficient amount of data from a real-world situation, which is not always available. In this regard, data augmentation is a popular technique used for data enrichment. The research conducted in this investigation established an insect pest dataset of common castor pests. This paper proposes a hybrid manipulation-based approach for data augmentation to solve the issue of the lack of a suitable dataset for effective vision-based model training. The deep convolutional neural networks VGG16, VGG19, and ResNet50 are then adopted to analyze the effects of the proposed augmentation method. The prediction results show that the proposed method addresses the challenges associated with adequate dataset size and significantly improves overall performance when compared to previous methods.