Drought is one of the most severe climatological disasters that has negative impact on agricultural production around the world. Over the years, computer vision technology has been used in conjunction with machine learning applications to replace traditional destructive and time-consuming methods for real-time monitoring of drought-affected plant. Deep learning (DL) techniques have gained a stellar reputation in image classification recently, with convolutional neural network (CNN) emerging as the industry standard. However, the size of deep CNN models is frequently large due to massive number of parameters and field application is often not feasible due to limited storage and computational resources. Several lightweight CNN models have been selected based on the number of network parameters of less than 6M and were trained and tested. The EfficientNet model has achieved a classification accuracy of 88.12 and 88.97 percent for identifying severe drought, mild drought, and no drought plants on visible and near-infrared images respectively. The findings of this study can be used to assist in the development of automated early detection of drought stressed plant with model sizes suitable for real-time plant diagnosis on mobile or embedded devices.