UAV-based visual inspection is frequently employed for surface defect recognition. However, the recognition accuracy of UAVs is diminished by the presence of background interference and the small size of defects. To address these challenges, this paper introduces a novel framework that comprises an online image preprocessing module and the Pipe-MobileNet neural-network-based model. The preprocessing module aims to generate images without background interference, while the Pipe-MobileNet model incorporates a customized depthwise convolution operator that classifies convolution kernels, making it more efficient in defect classification. To validate the effectiveness of the proposed method, a series of experiment was conducted on two realistic DN100 and DN200 pipelines. The results demonstrats that the peak signal-to-noise ratio (PSNR) reached 24.62, and the structural similarity index (SSIM) reached 0.94. In comparison with four other convolutional neural network models, the proposed method exhibited a notable enhancement in recognition accuracy, with an increase ranging from 30.1% to 56.9%. Additionally, the model training was sped up by 3%. These results underscore the method’s marked improvements in both recognition accuracy, as well as computational efficiency.