Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry

Gené-Mola, Jordi; Sanz, Ricardo; Rosell-Polo, Joan R.; Morros, Josep Ramon; Ruiz-Hidalgo, Javier; Vilaplana, Verónica; Gregorio, Eduard

doi:10.1016/j.compag.2019.105165

Cited by 125 publications

(32 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…DaSNet-V2 can efficiently and robustly perform visual sensing for automatic harvesting in apple orchards. Furthermore, Gené-Mola et al 68 performed a study on apple 2D detection with Mask R-CNN and 3D location utilizing structure-from-motion (SfM) photogrammetry. By testing 11 normally grown Fuji apple trees comprising a total number of 1455 apples, the system achieved encouraging performance with an F1-score that increased from 0.816 for 2D detection to 0.881 for 3D detection and location.…”

Section: Applications Of Deep Learning In Horticulture Cropsmentioning

confidence: 99%

Applications of deep-learning approaches in horticultural research: a review

Yang

2021

Hortic Res

View full text Add to dashboard Cite

Deep learning is known as a promising multifunctional tool for processing images and other big data. By assimilating large amounts of heterogeneous data, deep-learning technology provides reliable prediction results for complex and uncertain phenomena. Recently, it has been increasingly used by horticultural researchers to make sense of the large datasets produced during planting and postharvest processes. In this paper, we provided a brief introduction to deep-learning approaches and reviewed 71 recent research works in which deep-learning technologies were applied in the horticultural domain for variety recognition, yield estimation, quality detection, stress phenotyping detection, growth monitoring, and other tasks. We described in detail the application scenarios reported in the relevant literature, along with the applied models and frameworks, the used data, and the overall performance results. Finally, we discussed the current challenges and future trends of deep learning in horticultural research. The aim of this review is to assist researchers and provide guidance for them to fully understand the strengths and possible weaknesses when applying deep learning in horticultural sectors. We also hope that this review will encourage researchers to explore some significant examples of deep learning in horticultural science and will promote the advancement of intelligent horticulture.

show abstract

Section: Applications Of Deep Learning In Horticulture Cropsmentioning

confidence: 99%

Applications of deep-learning approaches in horticultural research: a review

Yang

2021

Hortic Res

View full text Add to dashboard Cite

show abstract

“…Up till now, there have been many studies in the aspect of apple targets recognition using deep learning technology. Many convolutional neural networks, such as YOLOv2 [19], YOLOv3 [20], LedNet [21], R-FCN [22], Faster R-CNN [23][24][25][26], Mask R-CNN [27], DaS-Net [28] and DaSNet-v2 [29], were successfully used in apple target recognition. The relevant study status is shown in Table 1.…”

Section: Introductionmentioning

confidence: 99%

A Real-Time Apple Targets Detection Method for Picking Robot Based on Improved YOLOv5

et al. 2021

View full text Add to dashboard Cite

The apple target recognition algorithm is one of the core technologies of the apple picking robot. However, most of the existing apple detection algorithms cannot distinguish between the apples that are occluded by tree branches and occluded by other apples. The apples, grasping end-effector and mechanical picking arm of the robot are very likely to be damaged if the algorithm is directly applied to the picking robot. Based on this practical problem, in order to automatically recognize the graspable and ungraspable apples in an apple tree image, a light-weight apple targets detection method was proposed for picking robot using improved YOLOv5s. Firstly, BottleneckCSP module was improved designed to BottleneckCSP-2 module which was used to replace the BottleneckCSP module in backbone architecture of original YOLOv5s network. Secondly, SE module, which belonged to the visual attention mechanism network, was inserted to the proposed improved backbone network. Thirdly, the bonding fusion mode of feature maps, which were inputs to the target detection layer of medium size in the original YOLOv5s network, were improved. Finally, the initial anchor box size of the original network was improved. The experimental results indicated that the graspable apples, which were unoccluded or only occluded by tree leaves, and the ungraspable apples, which were occluded by tree branches or occluded by other fruits, could be identified effectively using the proposed improved network model in this study. Specifically, the recognition recall, precision, mAP and F1 were 91.48%, 83.83%, 86.75% and 87.49%, respectively. The average recognition time was 0.015 s per image. Contrasted with original YOLOv5s, YOLOv3, YOLOv4 and EfficientDet-D0 model, the mAP of the proposed improved YOLOv5s model increased by 5.05%, 14.95%, 4.74% and 6.75% respectively, the size of the model compressed by 9.29%, 94.6%, 94.8% and 15.3% respectively. The average recognition speeds per image of the proposed improved YOLOv5s model were 2.53, 1.13 and 3.53 times of EfficientDet-D0, YOLOv4 and YOLOv3 and model, respectively. The proposed method can provide technical support for the real-time accurate detection of multiple fruit targets for the apple picking robot.

show abstract

“…Researchers used deep learning algorithms to segment and count different varieties of grapes, which could adapt to the complex environment of farmland and avoid the problem of multiple counting [ 19 , 20 ]. Gené-Mola et al [ 21 ] used instance segmentation network and dynamic structural photogrammetry technology to segment and locate apple images in the field. The results of fruit location in 3D point cloud showed that F 1 score was 0.88, which effectively reduced false positives in the process of recognition.…”

Section: Introductionmentioning

confidence: 99%

An Instance Segmentation-Based Method to Obtain the Leaf Age and Plant Centre of Weeds in Complex Field Environments

Quan

Mao

et al. 2021

Sensors

View full text Add to dashboard Cite

Leaf age and plant centre are important phenotypic information of weeds, and accurate identification of them plays an important role in understanding the morphological structure of weeds, guiding precise targeted spraying and reducing the use of herbicides. In this work, a weed segmentation method based on BlendMask is proposed to obtain the phenotypic information of weeds under complex field conditions. This study collected images from different angles (front, side, and top views) of three kinds of weeds (Solanum nigrum, barnyard grass (Echinochloa crus-galli), and Abutilon theophrasti Medicus) in a maize field. Two datasets (with and without data enhancement) and two backbone networks (ResNet50 and ResNet101) were replaced to improve model performance. Finally, seven evaluation indicators are used to evaluate the segmentation results of the model under different angles. The results indicated that data enhancement and ResNet101 as the backbone network could enhance the model performance. The F1 value of the plant centre is 0.9330, and the recognition accuracy of leaf age can reach 0.957. The mIOU value of the top view is 0.642. Therefore, deep learning methods can effectively identify weed leaf age and plant centre, which is of great significance for variable spraying.

show abstract

Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry

Cited by 125 publications

References 32 publications

Applications of deep-learning approaches in horticultural research: a review

Applications of deep-learning approaches in horticultural research: a review

A Real-Time Apple Targets Detection Method for Picking Robot Based on Improved YOLOv5

An Instance Segmentation-Based Method to Obtain the Leaf Age and Plant Centre of Weeds in Complex Field Environments

Contact Info

Product

Resources

About