Fruit Quantity and Ripeness Estimation Using a Robotic Vision System

Abstract

Accurate localisation of crop remains highly challenging in unstructured environments such as farms. Many of the developed systems still rely on the use of hand selected features for crop identification and often neglect the estimation of crop quantity and quality, which is key to assigning labor during farming processes. To alleviate these limitations we present a robotic vision system that can accurately estimate the quantity and quality of sweet pepper (Capsicum annuum L), a key horticultural crop. This system consists of three parts: detection, quality estimation, and tracking. Efficient detection is achieved using the FasterRCNN framework. Quality is then estimated in the same framework by learning a parallel layer which we show experimentally results in superior performance than treating quality as extra classes in the traditional Faster-RCNN framework. Evaluation of these two techniques outlines the improved performance of the parallel layer, where we achieve an F1 score of 77.3 for the parallel technique yet only 72.5 for the best scoring (red) of the multi-class implementation.To track the crop we present a tracking via detection approach, which uses the FasterRCNN with parallel layers, that is also a vision-only solution. This approach is cheap to implement as it only requires a camera and in experiments across 2 days we show that our proposed system can accurately estimate the number of sweet pepper present, within 4.1% of the ground truth.

Automated detection of koalas using low-level aerial surveillance and machine learning

“…To reduce repeated detections of the same koala, detected koalas were tracked over frames using a simple greedy tracking algorithm 38 . The result of this overall process was a set of koala detections that spanned multiple frames, which could be exported by the algorithm for manual review.…”

“…Although DCNNs typically require thousands or even millions of observations to train, a technique commonly referred to as fine-tuning allows this training cost to be greatly reduced 32,33,38 . Fine-tuning takes a network trained on a very large, general purpose dataset and adapts the model to new task by adapting the learnt weights on a small image dataset 32,33,38 .…”

“…The 918 images used represents significantly more images, and a similar number of overall detections as used in previous work 38 . Large numbers of negative examples were deliberately avoided to ensure that the model has maximum probability of detection.…”

Development of a sweet pepper harvesting robot

“…Due to the complex problem, most R&D on robotic harvesting focuses on a single aspect of the robotic system, for example, detection (Halstead, McCool, Denman, Perez, & Fookes, 2018;Kamilaris & Prenafeta-Boldú, 2018;Kapach, Barnea, Mairon, Edan, & Ben-Shahar, 2012;Vitzrabin & Edan, 2016a, 2016bZemmour, Kurtser, & Edan, 2019;Zhao, Gong, Huang, & Liu, 2016), manipulation and gripping (Bulanon & Kataoka, 2010;Eizicovits & Berman, 2014;Eizicovits, van Tuijl, Berman, & Edan, 2016;Rodríguez, Moreno, Sánchez, & Berenguel, 2013;Tian, Zhou, & Gu, 2018), and motion/task planning (Barth, IJsselmuiden, Hemming, & Van Henten, 2016; Korthals et al, 2018;Li & Qi, 2018;Liu, ElGeneidy, Pearson, Huda, & Neumann, 2018;.…”

Monocular Camera Based Fruit Counting and Mapping With Semantic Data Association

“…Das et al use a Support Vector Machine (SVM) to detect fruits, and use optical flow to associate the fruits in between subsequent images [1]. Halstead et al use a 2D tracking algorithm to track and refine Faster R-CNN detections of sweet peppers for counting and crop quantity evaluation in an indoor environment [13]. Roy et al develop a four-step 3D reconstruction method which first roughly aligns 3D point cloud two-side view of fruit tree row, then generates semantic representation with deep learning-based trunk segmentations and further refines two-view alignment with this data.…”