An innovative framework for peach tree disease recognition and segmentation is proposed in this paper, with the aim of significantly enhancing model performance in complex agricultural settings through deep learning techniques and data fusion strategies. The core innovations include a tiny feature attention mechanism backbone network, an aligned-head module, a Transformer-based semantic segmentation network, and a specially designed alignment loss function. The integration of these technologies not only optimizes the model’s ability to capture subtle disease features but also improves the efficiency of integrating sensor and image data, further enhancing the accuracy of the segmentation tasks. Experimental results demonstrate the superiority of this framework. For disease detection, the proposed method achieved a precision of 94%, a recall of 92%, and an accuracy of 92%, surpassing classical models like AlexNet, GoogLeNet, VGGNet, ResNet, and EfficientNet. In lesion segmentation tasks, the proposed method achieved a precision of 95%, a recall of 90%, and an mIoU of 94%, significantly outperforming models such as SegNet, UNet, and UNet++. The introduction of the aligned-head module and alignment loss function provides an effective solution for processing images lacking sensor data, significantly enhancing the model’s capability to process real agricultural image data. Through detailed ablation experiments, the study further validates the critical role of the aligned-head module and alignment loss function in enhancing model performance, particularly in the attention-head ablation experiment where the aligned-head configuration surpassed other configurations across all metrics, highlighting its key role in the overall framework. These experiments not only showcase the theoretical effectiveness of the proposed method but also confirm its practical value in agricultural disease management practices.