Pine wilt nematode disease is a devastating forest disease that spreads rapidly. Using drone remote sensing to monitor pine wilt nematode trees promptly is an effective way to control the spread of pine wilt nematode disease. In this study, the YOLOv4 algorithm was used to automatically identify abnormally discolored wilt from pine wilt nematode disease on UAV remote sensing images. Because the network structure of YOLOv4 is too complex, although the detection accuracy is high, the detection speed is relatively low. To solve this problem, the lightweight deep learning network MobileNetv2 is used to optimize the backbone feature extraction network. Furthermore, the YOLOv4 algorithm was improved by improving the backbone network part, adding CBAM attention, and adding the Inceptionv2 structure to reduce the number of model parameters and improve the accuracy and efficiency of identification. The speed and accuracy of the Faster R-CNN, YOLOv4, SSD, YOLOv5, and the improved MobileNetv2-YOLOv4 algorithm were compared, and the detection effects of the Faster R-CNN, YOLOv4, SSD, YOLOv5 and the improved MobileNetv2-YOLOv4 algorithm on trees with pine wilt nematode were analyzed. The experimental results show that the average precision of the improved MobileNetv2-YOLOv4 algorithm is 86.85%, the training time of each iteration cycle is 156 s, the parameter size is 39.23 MB, and the test time of a single image is 15 ms, which is better than Faster R-CNN, YOLOv4, and SSD, but comparable to YOLOv5. Compared with the advantages and disadvantages, comprehensively comparing these four indicators, the improved algorithm has a more balanced performance in the detection speed, the parameter size, and the average precision. The F1 score of the improved algorithm (95.60%) was higher than that of Faster R-CNN (90.80%), YOLOv4 (94.56%), and SSD (92.14%), which met the monitoring requirements of pine wilt nematode trees. Faster R-CNN and SSD pine-wilt-nematode tree detection models are not ideal in practical applications. Compared with the YOLOv4 pine-wilt-nematode tree detection model, the improved MobileNetv2-YOLOv4 algorithm satisfies the condition of maintaining a lower model parameter quantity to obtain higher detection accuracy; therefore, it is more suitable for practical application scenarios of embedded devices. It can be used for the rapid detection of pine wilt nematode diseased trees.