Faster R–CNN–based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting

Fu, Longsheng; Majeed, Yaqoob; Zhang, Xin; Karkee, Manoj; Zhang, Qin

doi:10.1016/j.biosystemseng.2020.07.007

Cited by 162 publications

(72 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, DeepSeedling integrated Faster R-CNN as the target detection network, then developed a framework to track and count cotton seedlings in the videos [ 17 ]. In addition, Faster R-CNN was used as the detection network for detecting other agricultural objects such as plant leaves [ 31 ], maize seedlings [ 32 ], kiwi fruit [ 33 ], grapes [ 34 ], and apples [ 35 ]. However, most Faster-R-CNN-based approaches are offline.…”

Section: Introductionmentioning

confidence: 99%

A Convolutional Neural Network-Based Method for Corn Stand Counting in the Field

Wang

Xiang

Tang

et al. 2021

Sensors

View full text Add to dashboard Cite

Accurate corn stand count in the field at early season is of great interest to corn breeders and plant geneticists. However, the commonly used manual counting method is time consuming, laborious, and prone to error. Nowadays, unmanned aerial vehicles (UAV) tend to be a popular base for plant-image-collecting platforms. However, detecting corn stands in the field is a challenging task, primarily because of camera motion, leaf fluttering caused by wind, shadows of plants caused by direct sunlight, and the complex soil background. As for the UAV system, there are mainly two limitations for early seedling detection and counting. First, flying height cannot ensure a high resolution for small objects. It is especially difficult to detect early corn seedlings at around one week after planting, because the plants are small and difficult to differentiate from the background. Second, the battery life and payload of UAV systems cannot support long-duration online counting work. In this research project, we developed an automated, robust, and high-throughput method for corn stand counting based on color images extracted from video clips. A pipeline developed based on the YoloV3 network and Kalman filter was used to count corn seedlings online. The results demonstrate that our method is accurate and reliable for stand counting, achieving an accuracy of over 98% at growth stages V2 and V3 (vegetative stages with two and three visible collars) with an average frame rate of 47 frames per second (FPS). This pipeline can also be mounted easily on manned cart, tractor, or field robotic systems for online corn counting.

show abstract

Section: Introductionmentioning

confidence: 99%

A Convolutional Neural Network-Based Method for Corn Stand Counting in the Field

Wang

Xiang

Tang

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Because interrow spacing is 2.7–3.8 m, a depth threshold of 1.4–1.9 m (half of the row spacing) was used to remove objects from the adjacent rows and create a foreground image (Figure 4c). Our preliminary study showed that by filtering out the image background, the accuracy in object detection could be improved by ~2.5%, in general, compared to the images with background using deep learning techniques (Fu et al, 2020). The images were processed using a MATLAB (R2018b) software package on a Windows 10 (64‐bit) platform with Intel Core i7‐8750H CPU (2.20 GHz, 32.0 GB RAM, NVIDIA GeForce GTX 1080 GPU with Max‐Q design).…”

Section: Methodsmentioning

confidence: 99%

Computer vision‐based tree trunk and branch identification and shaking points detection in Dense‐Foliage canopy for automated harvesting of apples

Zhang

Karkee

Zhang

et al. 2020

Journal of Field Robotics

Self Cite

View full text Add to dashboard Cite

Fresh market apples are one of the high-value crops in the United States.Washington alone has produced two-thirds of the annual national production in the past 10 years. However, the availability of seasonal labor is increasingly uncertain. Shake-and-catch automated harvesting solutions have, therefore, become attractive for addressing this challenge. One of the significant challenges in applying this harvesting system is effectively positioning the end-effector at appropriate excitation locations. A computer vision system was used for automatically identifying appropriate locations. Convolutional neural networks (CNNs) were utilized to identify the tree trunks and branches for supporting the automated excitation locations determination. Three CNN architectures were employed: Deeplab v3+ ResNet-18, VGG-16, and VGG-19. Four pixel classes were predefined as branches, trunks, apples, and leaves to segment the canopies trained to simple, narrow, accessible, and productive tree architectures with varying foliage density. Results on Fuji cultivar showed that ResNet-18 outperformed the VGGs in identifying branches and trunks based on all three evaluation measures: per-class accuracy (PcA), intersection over union (IoU), and boundary-F1 score (BFScore). ResNet-18 achieved a PcA of 97%, IoU of 0.69, and BFScore of 0.89. The ResNet-18 was further evaluated for its robustness with other test canopy images. When applied this method to one of the highest density cultivars of Scifresh, results showed it can achieve IoUs of 0.41 and 0.62 and BFScores of 0.71 and 0.86 for branches and trunks. Such identification result helped to get a 72% of auto-determined shaking points being the "good" category identified by human experts.

show abstract

“…The progress of transfer learning, a technique that allows the use of pretrained SOTA CNNs as base models in DL, and the availability of public DL libraries have contributed to the exponential adoption of DL in plant phenotyping. Deep CNN approaches for imagebased plant phenotyping have been applied for plant stress evaluation, plant development characterisation, crop postharvest quality assessment, and fruit detection and yield evaluation [4,[10][11][12][13][14][15]. However, not all modern CNN solutions can be readily implemented for plant phenotyping applications and adoption will require extra efforts, which may be technically challenging [4].…”

Section: Introductionmentioning

confidence: 99%

Automated Machine Learning for High-Throughput Image-Based Plant Phenotyping

2021

View full text Add to dashboard Cite

Automated machine learning (AutoML) has been heralded as the next wave in artificial intelligence with its promise to deliver high-performance end-to-end machine learning pipelines with minimal effort from the user. However, despite AutoML showing great promise for computer vision tasks, to the best of our knowledge, no study has used AutoML for image-based plant phenotyping. To address this gap in knowledge, we examined the application of AutoML for image-based plant phenotyping using wheat lodging assessment with unmanned aerial vehicle (UAV) imagery as an example. The performance of an open-source AutoML framework, AutoKeras, in image classification and regression tasks was compared to transfer learning using modern convolutional neural network (CNN) architectures. For image classification, which classified plot images as lodged or non-lodged, transfer learning with Xception and DenseNet-201 achieved the best classification accuracy of 93.2%, whereas AutoKeras had a 92.4% accuracy. For image regression, which predicted lodging scores from plot images, transfer learning with DenseNet-201 had the best performance (R2 = 0.8303, root mean-squared error (RMSE) = 9.55, mean absolute error (MAE) = 7.03, mean absolute percentage error (MAPE) = 12.54%), followed closely by AutoKeras (R2 = 0.8273, RMSE = 10.65, MAE = 8.24, MAPE = 13.87%). In both tasks, AutoKeras models had up to 40-fold faster inference times compared to the pretrained CNNs. AutoML has significant potential to enhance plant phenotyping capabilities applicable in crop breeding and precision agriculture.

show abstract

Faster R–CNN–based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting

Cited by 162 publications

References 44 publications

A Convolutional Neural Network-Based Method for Corn Stand Counting in the Field

A Convolutional Neural Network-Based Method for Corn Stand Counting in the Field

Computer vision‐based tree trunk and branch identification and shaking points detection in Dense‐Foliage canopy for automated harvesting of apples

Automated Machine Learning for High-Throughput Image-Based Plant Phenotyping

Contact Info

Product

Resources

About