Early detection of plant nutrient deficiency is crucial for agricultural productivity. This study investigated the performance and interpretability of Convolutional Neural Networks (CNNs) for this task. Using the rice and banana datasets, we compared three CNN architectures (CNN, VGG-16, Inception-V3). Inception-V3 achieved the highest accuracy (93% for rice and banana), but simpler models such as VGG-16 might be easier to understand. To address this trade-off, we employed Explainable AI (XAI) techniques (SHAP and Grad-CAM) to gain insights into model decision-making. This study emphasises the importance of both accuracy and interpretability in agricultural AI and demonstrates the value of XAI for building trust in these models.