Automatic Detection of Single Ripe Tomato on Plant Combining Faster R-CNN and Intuitionistic Fuzzy Set

Abstract

Fast and accurate detection of ripe tomatoes on plant, which replaces manual labor with a robotic vision-based harvesting system, is a challenging task. Tomatoes in adjacent positions are easily mistaken as a single tomato by image recognition methods. In this study, a ripe tomato detection method that combines deep learning with edge contour detection is proposed. Our approach efficiently separates target tomatoes from overlapping tomatoes to detect individual fruits. This approach yields several improvements. First, deep learning requires less time and extracts deeper features than traditional methods for assessing candidate ripe tomato regions. Second, we use Gaussian density function of H and S in the HSV color space to help segment tomato regions from the background, followed by erosion and dilation on the tomato body to separate adjacent tomatoes and remove peripheral subpixels from all detected ripe tomatoes. Third, an adaptive threshold intuitionistic fuzzy set (IFS) method was developed to identify the tomato's edge, and it performs well in detecting blurred edges in overlapping regions. To improve the efficiency and stability of edge detection under natural conditions, we adopted an illumination adjustment algorithm for the tomato image before edge detection. As samples, we collected images showing tomatoes that were separated, adjacent, overlapped, and even shaded by leaves. The widths and heights of these tomato samples were calculated and analyzed to evaluate the detection performance of the proposed method. The root mean square error (RMSE) results for tomato width and height using the proposed method are 2.996 pixels and 3.306 pixels, respectively. The mean relative error percent (MRE%) values for horizontal and vertical center position shift are 0.261% and 1.179%, respectively. These results demonstrate that the proposed method improves tomato detection accuracy and that it can be further applied in the harvesting process of agricultural robots. INDEX TERMS Tomato detection, deep learning, background subtraction, intuitionistic fuzzy set theory (IFS), contour segmentation. XUAN LIU received the B.S. degree in electrical engineering and automation, Nanjing Forestry University, China, in 2018, where he is currently pursuing the M.S. degree in instrument engineering. His research interests include computer vision and machine learning. ZHOU PAN received the B.S. degree in electrical engineering and automation from the Huaian Institute of Technology, China, in 2016. She is currently pursuing the M.S. degree in control theory and control engineering with Nanjing Forestry University, China. Her research interests include computer vision and machine learning.

Applications of deep-learning approaches in horticultural research: a review

“…From the brief introduction of several commonly used models described above, we know that deep-learning technology has a powerful role in image classification, regression and segmentation. Furthermore, there are also many other kinds of network structures that are applied, such as single shot multibox detection (SSD) 56 , long short-term memory (LSTM) 32 , you only look once (YOLO, source code is available at: http://pjreddie.com/yolo/ ) 57 , regions-CNN (R-CNN) 58 , fast region-based CNNs (Fast R-CNN, source code is available at: https://arxiv.org/abs/1504.08083 ) 33 , faster region-based CNNs (Faster-RCNN, source code is available at: https://github.com/shaoqingren/faster_rcnn (in MATLAB) and at https://github.com/rbgirshick/py-faster-rcnn (in Python)) 44 , 59 , 60 , and so on. In addition, the processed data types are not simply limited to RGB images but can also include any other data forms, such as video, hyperspectral images, and spectral data.…”

“…In another study, Sun et al 73 used a CNN based on the original Faster R-CNN algorithm to detect and identify flowers and mature (red) and immature (green) fruits of tomatoes. Hu et al 60 introduced a method that combined intuitionistic fuzzy set (IFS) theory with the Faster R-CNN model to detect individual ripe tomatoes. The ripe tomato image dataset, which includes adjacent, separated, leaf-shaded, and overlapped images, was used to obtain exact values of the height and width, and these data were then analyzed to evaluate the overall performance of the proposed detection model.…”

Stochastic Decision Fusion of Convolutional Neural Networks for Tomato Ripeness Detection in Agricultural Sorting Systems

“…Liu et al [ 12 ] studied an algorithm combining a coarse-to-fine scanning method and a false-color removal method to detect mature tomatoes. To achieve accurate ripe tomato detection, Hu et al [ 13 ] suggested a method that combines a deep learning algorithm and an edge-contour analysis method. Sun et al [ 14 ] proposed an improved feature-pyramid-network-based tomato organ recognition method.…”

An Intelligent Self-Service Vending System for Smart Retail

“…Traditional image recognition mainly includes four links: image acquisition, image preprocessing, feature extraction, and pattern matching, while the main complex work focuses on the image preprocessing and feature extraction stages. The authors of [27] proposed a detection method that combines Fast R-CNN with an adaptive threshold intuitionistic fuzzy set to segment and recognizes overlapping or occluded tomatoes on plants.…”