Pest detection and identification of diseases in agricultural crops is essential to ensure good product since it is the major challenge in the field of agriculture. Therefore, effective measures should be taken to fight the infestation to minimise the use of pesticides. The techniques of image analysis are extensively applied to agricultural science that provides maximum protection to crops. This might obviously lead to better crop management and production. However, automatic pest detection with machine learning technology is still in the infant stage. Hence, the video processing-based pest detection framework is constructed in this work by following six major phases, viz. (a) Video Frame Acquisition, (b) Pre-processing, (c) Object Tracking, (d) Foreground and Background Segmentation, (e) Feature Extraction, and (f) Classification. Initially, the moving frames of videos are pre-processed, and the movement of the object is tracked with the aid of the foreground and background segmentation approach via K-Means clustering. From the segmented image, a new feature evaluation termed as Distributed Intensity-based LBP features (DI-LBP) along with edges and colour are extracted. Further, the features are subjected to a classification process, where an optimised Neural Network (NN) is used. As a novelty, the training of NN will be carried out using a new Dragonfly with New Levy Update (D-NU) algorithm via updating the weight. Finally, the performance of the proposed model is analysed over other conventional models with respect to certain performance measures for both video and image datasets.