Culling Double Counting in Sequence Images for Fruit Yield Estimation

Abstract

Exact yield estimation of fruits on plants guaranteed fine and timely decisions on harvesting and marketing practices. Automatic yield estimation based on unmanned agriculture offers a viable solution for large orchards. Recent years have witnessed notable progress in computer vision with deep learning for yield estimation. Yet, the current practice of vision-based yield estimation with successive frames may engender fairly great error because of the double counting of repeat fruits in different images. The goal of this study is to provide a wise framework for fruit yield estimation in sequence images. Specifically, the anchor-free detection architecture (CenterNet) is utilized to detect fruits in sequence images from videos collected in the apple orchard and orange orchard. In order to avoid double counts of a single fruit between different images in an image sequence, the patch matching model is designed with the Kuhn–Munkres algorithm to optimize the paring process of repeat fruits in a one-to-one assignment manner for the sound performance of fruit yield estimation. Experimental results show that the CenterNet model can successfully detect fruits, including apples and oranges, in sequence images and achieved a mean Average Precision (mAP) of 0.939 under an IoU of 0.5. The designed patch matching model obtained an F1-Score of 0.816 and 0.864 for both apples and oranges with good accuracy, precision, and recall, which outperforms the performance of the reference method. The proposed pipeline for the fruit yield estimation in the test image sequences agreed well with the ground truth, resulting in a squared correlation coefficient of R2apple = 0.9737 and R2orange = 0.9562, with a low Root Mean Square Error (RMSE) for these two varieties of fruit.

Approach of Dynamic Tracking and Counting for Obscured Citrus in Smart Orchard Based on Machine Vision

“…Considering the practical applications, Xia et al [17] achieved the detection and automatic estimation of citrus with a lightweight model for edge computing devices and Zhang et al [28] used the Hough's circle transform method on MATLAB software (v. 2017b) to achieve rapid automatic fruit detection and yield prediction. These approaches are difficult to port to embedded devices and require specialized edge computing devices.…”

“…ZHANG et al [16] proposed the "OrangeSort" method based on simple online and realtime tracking (SORT) for counting citrus in the field, which reduces the effect of leaf shading on the accuracy of citrus counting results, but the method ignores the appearance characteristics of the target. Xia et al [17] introduced the Byte algorithm to improve FairMOT's data association strategy during fruit tracking to predictively track fruits in videos, but its use of depth features only in appearance does not fully represent the target and affects the tracking accuracy. Wojke et al [18] proposed the DeepSort method which incorporates both motion and appearance information and which can achieve high multitarget tracking accuracy (MOTA) while maintaining real-time speed, and can effectively reduce the impact of the occlusion problem.…”

Smart Fruit Growing Through Digital Twin Paradigm: Systematic Review and Technology Gap Analysis

“…In the meantime, Gao et al (2022) and Fu et al (2022) presented research suitable for fruit harvesting: (1) real-time fruit detection from video using YOLOv4-tiny and Kinect V2; and 2) detection of banana stalks using YOLOv4. Meantime, Xia et al (2022) applied CenterNet, which detects centre-points of objects, and the Kuhn-Munkres algorithm for object tracking. Anderson et al (2021) analysed autonomous estimation of fruit load in complex terms of business processes in the field: distance to a tree, daytime, cultivars, and others, completing an experiment using a prototype of a UGV.…”

Fast region based convolutional neural network ResNet-50 model for on tree Mango fruit yield estimation

“…The methods used to estimate tree fruit yields include visual inspection of the state of the trees, manual fruit counting, and knowledge of prior production histories. The current best approach for estimating fruit load involves manually counting a sample of trees, however, this is labor-intensive and occasionally inaccurate [2]. The exact pixel-by-pixel instance segmentation of fruit and the proper identification of picking sites are two challenging tasks that the vision system of a fruitpicking robot must do.…”