Crop diseases contribute significantly to food insecurity, malnutrition, and poverty in Africa where the majority of the population is into Agriculture. Manual plant disease recognition methods are widespread but limited, ineffective, costly, and time-consuming making the need to search for automatic and efficient methods of recognition more crucial. Machine learning and Convolutional Neural Networks have been applied in other jurisdictions in an attempt to solve these problems. They have achieved impressive results in this domain but tend to be 'data-hungry', invariant, and vulnerable to attacks that can easily lead to misclassifications. Capsule Networks, on the other hand, avoids the weaknesses of CNNs and has not been widely used in this area. This article, therefore, proposes the use of Gabor and Capsule network to recognize blurred, deformed, and unseen tomato and citrus disease images. Experimental results show that the proposed model can achieve a 98.13% test accuracy which is comparable to the performance of state-of-theart CNN models in the literature. Also, the proposed model outperformed two state-of-the-art deep learning models (which were implemented as baseline models) in terms of robustness, flexibility, fast converges, and having fewer parameters. This work can be extended to other crops and may well serve as a useful tool for the recognition of unseen plant diseases under bad weather and bad illumination conditions.