Due to its high potential and value, the Internet of things (IoT) has been used in various areas such as information security, industry 4.0, and smart agriculture. IoT is used in agriculture through the use of sensors, unmanned aerial vehicles (UAV), satellite technologies, robots, image processing, and artificial intelligence technologies. These smart agricultural practices increase production and quality and lead to savings in irrigation, thereby reducing environmental pollution during production. This study proposes an ultra-lightweight automated plant species classification method for smart agriculture applications. A UAV is used to acquire a new image dataset. An ultra-lightweight classification method is then used to classify the acquired plant species images. Our proposed ultra-lightweight computer vision model presents a histogram-based simple feature extraction function. The presented feature extractor uses histogram extraction and median filter in conjunction. The generated features are fed to two shallow classifiers, which are the support vector machine (SVM), and k nearest neighbor (KNN). The utilized SVM and KNN classifiers have attained 96.45% and 94.11% accuracies consecutively. The results demonstrate that this model is very capable of plant image classification and is ready for use in a physical agriculture environment.