The correct diagnosis and recognition of crop diseases play an important role in ensuring crop yields and preventing food safety. The existing methods for crop disease recognition mainly focus on accuracy while ignoring the algorithm’s robustness. In practice, the acquired images are often accompanied by various noises. These noises lead to a huge challenge for improving the robustness and accuracy of the recognition algorithm. In order to solve this problem, this paper proposes a residual self-calibration and self-attention aggregation network (RCAA-Net) for crop disease recognition in actual scenarios. The proposed RCAA-Net is composed of three main modules: (1) multi-scale residual module, (2) feedback self-calibration module, and (3) self-attention aggregation module. Specifically, the multi-scale residual module is designed to learn multi-scale features and provide both global and local information for the appearance of the disease to improve the performance of the model. The feedback self-calibration is proposed to improve the robustness of the model by suppressing the background noise in the original deep features. The self-attention aggregation module is introduced to further improve the robustness and accuracy of the model by capturing multi-scale information in different semantic spaces. The experimental results on the challenging 2018ai_challenger crop disease recognition dataset show that the proposed RCAA-Net achieves state-of-the-art performance on robustness and accuracy for crop disease recognition in actual scenarios.