Date Fruit Classification for Robotic Harvesting in a Natural Environment Using Deep Learning

Abstract

An accurate vision system to classify and analyze fruits in real time is critical for harvesting robots to be cost-effective and efficient. However, practical success in this area is still limited, and to the best of our knowledge, there is no research in the area of machine vision for date fruits in an orchard environment. In this work, we propose an efficient machine vision framework for date fruit harvesting robots. The framework consists of three classification models used to classify date fruit images in real time according to their type, maturity, and harvesting decision. In the classification models, deep convolutional neural networks are utilized with transfer learning and fine-tuning on pre-trained models. To build a robust vision system, we create a rich image dataset of date fruit bunches in an orchard that consists of more than 8000 images of five date types in different pre-maturity and maturity stages. The dataset has a large degree of variations that reflects the challenges in the date orchard environment including variations in angles, scales, illumination conditions, and date bunches covered by bags. The proposed date fruit classification models achieve accuracies of 99.01%, 97.25%, and 98.59% with classification times of 20.6, 20.7, and 35.9 msec for the type, maturity, and harvesting decision classification tasks, respectively. INDEX TERMS Dates classification, maturity analysis, automated harvesting, deep learning, convolutional neural networks.

IHDS: Intelligent Harvesting Decision System for Date Fruit Based on Maturity Stage Using Deep Learning and Computer Vision

“…The comparison will be based on well-known performance metrics (F1 score, accuracy, sensitivity (recall), and precision). Our study and a reference study by Altaheri, H., M. Alsulaiman, et al [13] used the same datasets in a farm environment and the date fruit bunches in an orchard, whereas other studies used different datasets using single dates with uniform background. Table 7 illustrates a comparison of the evaluation parameters of the proposed system and the reference study of Nasiri, A., A. Taheri-Garavand, et al [12].…”

“…They collected the dataset through a smartphone, and their system achieved an overall accuracy of 96.98%. Another study has been done by Altaheri, H., M. Alsulaiman, et al [13], who proposed a framework using a vision system to classify date fruits in an orchard environment. They used the proposed framework to classify date fruit images based on type and maturity.…”

Smart Camera for Quality Inspection and Grading of Food Products

“…The majority of work related to date processing is for date classification or recognition. CNN was used for assisting consumers to identify the variety and origination of the dates [28] and classifying dates according to their type and maturity level for robotic harvest decisions [29]. Basic image processing techniques were used for grading date maturity in HSV (Hue, Saturation and Value) color space [30].…”

Date fruit dataset for intelligent harvesting

“…The labeling instructions and rules are explained in detail in Ref. [2].

Fig. 2Sample images of a Sullaj date captured in six imaging sessions and covered all date maturity stages (immature, Khalal, Rutab, and Tamar).

…”