Kiwifruit detection in field images using Faster R-CNN with ZFNet

Fu, Longsheng; Feng, Yali; Majeed, Yaqoob; Zhang, Xin; Zhang, Jing; Karkee, Manoj; Zhang, Qin

doi:10.1016/j.ifacol.2018.08.059

Cited by 99 publications

(48 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, convolutional neural networks (CNN) showed superior performance in object detection applications [13]- [15]. There have been some researches using different CNN architectures for fruit detection such as apple [16]- [18], mango [19], strawberry [20] and kiwifruit [21], [22]. Bargoti and Underwood [23] applied VGG16 to detect mangoes and apples in orchards, which achieved an F1-score of 0.9.…”

Section: Introductionmentioning

confidence: 99%

“…Fuet al [21] used LeNet to detect kiwifruits in the orchard, which reached 89.29% detection rate and cost 0.27 s on average to recognize a fruit. Fu et al [22] presented a kiwifruit image detection using ZFNet and achieved 92.3% detection rate and cost 0.005 s on average to detect a fruit. These studies showed good promising for fruit detection in RGB images using CNN.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improved Kiwifruit Detection Using Pre-Trained VGG16 With RGB and NIR Information Fusion

Liu

et al. 2020

IEEE Access

Self Cite

140

View full text Add to dashboard Cite

This study presents a novel method to apply the RGB-D (Red Green Blue-Depth) sensors and fuse aligned RGB and NIR images with deep convolutional neural networks (CNN) for fruit detection. It aims to build a more accurate, faster, and more reliable fruit detection system, which is a vital element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). A common Faster R-CNN network VGG16 was adopted through transfer learning, for the task of kiwifruit detection using imagery obtained from two modalities: RGB (red, green, blue) and Near-Infrared (NIR) images. Kinect v2 was used to take a bottom view of the kiwifruit canopy's NIR and RGB images. The NIR (1 channel) and RGB images (3 channels) were aligned and arranged side by side into a 6-channel image. The input layer of the VGG16 was modified to receive the 6-channel image. Two different fusion methods were used to extract features: Image-Fusion (fusion of the RGB and NIR images on input layer) and Feature-Fusion (fusion of feature maps of two VGG16 networks where the RGB and NIR images were input respectively). The improved networks were trained end-to-end using back-propagation and stochastic gradient descent techniques and compared to original VGG16 networks with RGB and NIR image input only. Results showed that the average precision (APs) of the original VGG16 with RGB and NIR image input only were 88.4% and 89.2% respectively, the 6-channel VGG16 using the Feature-Fusion method reached 90.5%, while that using the Image-Fusion method reached the highest AP of 90.7% and the fastest detection speed of 0.134 s/image. The results indicated that the proposed kiwifruit detection approach shows a potential for better fruit detection.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Improved Kiwifruit Detection Using Pre-Trained VGG16 With RGB and NIR Information Fusion

Liu

et al. 2020

IEEE Access

Self Cite

140

View full text Add to dashboard Cite

show abstract

“…Deep learning, which can autonomously extract fruit features, has shown results in strawberry detection (Habaragamuwa et al, ). In addition to strawberries, deep learning, especially the Faster RCNN network, has been widely used for detection of many other fruits, including sweet pepper, mango, apple, almond, and kiwifruit (Fu et al, ; Mai, Zhang, & Meng, ; Sa et al, ; Zhang et al, ). All these systems used detection networks to generate bounding boxes around the target fruits.…”

Section: Related Workmentioning

confidence: 99%

An autonomous strawberry‐harvesting robot: Design, development, integration, and field evaluation

Xiong

Grimstad

et al. 2019

Journal of Field Robotics

292

133

View full text Add to dashboard Cite

This paper presents an autonomous robot capable of picking strawberries continuously in polytunnels. Robotic harvesting in cluttered and unstructured environment remains a challenge. A novel obstacle‐separation algorithm was proposed to enable the harvesting system to pick strawberries that are located in clusters. The algorithm uses the gripper to push aside surrounding leaves, strawberries, and other obstacles. We present the theoretical method to generate pushing paths based on the surrounding obstacles. In addition to manipulation, an improved vision system is more resilient to lighting variations, which was developed based on the modeling of color against light intensity. Further, a low‐cost dual‐arm system was developed with an optimized harvesting sequence that increases its efficiency and minimizes the risk of collision. Improvements were also made to the existing gripper to enable the robot to pick directly into a market punnet, thereby eliminating the need for repacking. During tests on a strawberry farm, the robots first‐attempt success rate for picking partially surrounded or isolated strawberries ranged from 50% to 97.1%, depending on the growth situations. Upon an additional attempt, the pick success rate increased to a range of 75–100%. In the field tests, the system was not able to pick a target that was entirely surrounded by obstacles. This failure was attributed to limitations in the vision system as well as insufficient dexterity in the grippers. However, the picking speed improved upon previous systems, taking just 6.1 s for manipulation operation in the one‐arm mode and 4.6 s in the two‐arm mode.

show abstract

“…False positives were observed in the pictures that corresponded to immature fruits (fruits green), and where the brightness of sunlight was slightly greater, and in those pictures that suffered from rolling shutter. These results can be significantly improved by taking pictures several times throughout the day, as suggested by Fu et al (2018), or by flying the UAV at a low speed. Finally, the F1-score exhibited values greater than 87%, indicating the high robustness of the trained model.…”

Section: Distribution Of Fruits In An Apple Orchard Canopymentioning

confidence: 99%

A Cloud-Based Environment for Generating Yield Estimation Maps From Apple Orchards Using UAV Imagery and a Deep Learning Technique

et al. 2020

View full text Add to dashboard Cite

Farmers require accurate yield estimates, since they are key to predicting the volume of stock needed at supermarkets and to organizing harvesting operations. In many cases, the yield is visually estimated by the crop producer, but this approach is not accurate or time efficient. This study presents a rapid sensing and yield estimation scheme using offthe-shelf aerial imagery and deep learning. A Region-Convolutional Neural Network was trained to detect and count the number of apple fruit on individual trees located on the orthomosaic built from images taken by the unmanned aerial vehicle (UAV). The results obtained with the proposed approach were compared with apple counts made in situ by an agrotechnician, and an R 2 value of 0.86 was acquired (MAE: 10.35 and RMSE: 13.56). As only parts of the tree fruits were visible in the top-view images, linear regression was used to estimate the number of total apples on each tree. An R 2 value of 0.80 (MAE: 128.56 and RMSE: 130.56) was obtained. With the number of fruits detected and tree coordinates two shapefile using Python script in Google Colab were generated. With the previous information two yield maps were displayed: one with information per tree and another with information per tree row. We are confident that these results will help to maximize the crop producers' outputs via optimized orchard management.

show abstract

Kiwifruit detection in field images using Faster R-CNN with ZFNet

Cited by 99 publications

References 5 publications

Improved Kiwifruit Detection Using Pre-Trained VGG16 With RGB and NIR Information Fusion

Improved Kiwifruit Detection Using Pre-Trained VGG16 With RGB and NIR Information Fusion

An autonomous strawberry‐harvesting robot: Design, development, integration, and field evaluation

A Cloud-Based Environment for Generating Yield Estimation Maps From Apple Orchards Using UAV Imagery and a Deep Learning Technique

Contact Info

Product

Resources

About