Fruit Localization and Environment Perception for Strawberry Harvesting Robots

Abstract

This work presents a machine vision system for the localization of strawberries and environment perception in a strawberry-harvesting robot for use in table-top strawberry production. A deep convolutional neural network for segmentation is utilized to detect the strawberries. Segmented strawberries are localized through coordinate transformation, density base point clustering and the proposed location approximation method. To avoid collisions between the gripper and fixed obstacles, the safe manipulation region is limited to the space in front of the table and underneath the strap. Therefore, a safe region classification algorithm, based on Hough Transform algorithm, is proposed to segment the strap masks into a belt region in order to identify the pickable strawberries located underneath the strap. Similarly, a safe region classification algorithm is proposed for the table, to calculate its points in 3D and fit the points onto a 3D plane based on the 3D point cloud, so that pickable strawberries in front of the table can be identified. Experimental tests showed that the algorithm could accurately classify ripe and unripe strawberries and could identify whether the strawberries are within the safe region for harvesting. Furthermore, harvester robot's optimized localization method could accurately locate the strawberry targets with a picking accuracy rate of 74.1% in modified situations. INDEX TERMS Robotics and automation, strawberry harvester, machine vision, environment perception.

L*a*b*Fruits: A Rapid and Robust Outdoor Fruit Detection System Combining Bio-Inspired Features with One-Stage Deep Learning Networks

“…Yu et al [34] 1900 Side on (Close) Chen et al [35] 12526 Aerial Lamb and Chuah [36] 4550 Ground Ge et al [37] -Side on Sa et al [8] 122…”

Visual Perception and Modeling for Autonomous Apple Harvesting

“…Moreover, based on the size of the object boundary boxes, the objects without the sufficient number of points or with severely in-balance length in different axis (X, Y, Z) will be deleted from object list (as shown in (b) and (c) in Figure 9). In the recent work of robotic harvesting of strawberry [14], similar issues of the depth sensing are reported. Their work applied a Density-Based Spatial Clustering (DBSC) to process the point clouds.…”

“…In the machine vision in agriculture applications, the works of [11] and [12] applied faster-RCNN to perform detection of multiple classes of fruits in farm conditions, including apples, mango, and pepper. The works of [13] and [14] utilised the Mask-RCNN model to perform detection and instance segmentation in the application of robotic harvesting of strawberry fruits. The works of [15] and [16] applied YOLO on real-time in-field detection of apple and mango fruits for yield estimation and monitoring, respectively.…”

Object Localization and Depth Estimation for Eye-in-Hand Manipulator Using Mono Camera

“…Choi et al [8] used CNN for the detection of various objects to be grabbed by the soft gripper. Ge et al [9] applied a faster detection capability using R-CNN for strawberry picking robots by localizing three environments first. Another applied Faster R-CNN has been reported to detect and localize objects (e.g., pallets) based on 2D rangefinder data [10].…”

“…The Light Detection and Ranging (LiDAR) and environmental imaging of outdoor vehicles are incorporated by Reina et al [25]. Ge et al [9] retained the traditional Hough Transform image processing technique to recognize the classifications of strawberry environments. References [7], [9], [26]- [28] also used RGB-D images to estimate the targets and sense obstacles in each environment.…”