Plant disease is a significant health concern among all living creatures. Early diagnosis can help farmers takenecessary steps to cure the disease and accelerate the production rate efficiently. Our research has beenconducted with five most common rice leaf diseases, such as bacterial leaf blight, brown spot, leaf blast,leaf scald, and narrow brown spot, including healthy class, and two categories of betel leaf, such as healthyand unhealthy class. A robust new deep ensemble model, based on InceptionResNetV2, EfficientNetV2L,and Xception, has been proposed, known as, PlantDet, in this research. PlantDet solves not only underfitting problems but also leverage nourished performances simultaneously for scarce dataset of the sparse number of different background image dataset. PlantDet integrates efficient data augmentation, preprocessing, Global Average Pooling layer, Dropout mechanism, L2 regularizers, PRelu activation function, Batch Normalization layers, and more Dense layers that make the model more robust compared to all existing models and help to handle underfitting and overfitting problems while maintaining high performance. PlantDet exceeds the previous state-of-art model for the Rice Leaf dataset with an Accuracy of 98.53%, a Precision of 98.50%, a Recall of 98.35%, a F1 of 98.42% and a Specificity of 99.71%. In addition, for the Betel Leaf dataset, PlantDet also surpassed all existing base models, including several robust ensemble models. Finally, Grad-CAM and Score-CAM have been accomplished with the Xception method to explain the model performances particularly to elaborate how the Deep Learning (DL) models works for this complex dataset. Score-CAM slightly outperformed Grad-CAM++ in terms of localizing the predicted area.