3D‐vision based detection, localization, and sizing of broccoli heads in the field

Abstract

This paper describes a 3D vision system for robotic harvesting of broccoli using low-cost RGB-D sensors, which was developed and evaluated using sensory data collected under real-world field conditions in both the UK and Spain. The presented method addresses the tasks of detecting mature broccoli heads in the field and providing their 3D locations relative to the vehicle. The paper evaluates different 3D features, machine learning, and temporal filtering methods for detection of broccoli heads. Our experiments show that a combination of Viewpoint Feature Histograms, Support Vector Machine classifier, and a temporal filter to track the detected heads results in a system that detects broccoli heads with high precision. We also show that the temporal filtering can be used to generate a 3D map of the broccoli head positions in the field. Additionally, we present methods for automatically estimating the size of the broccoli heads, to determine when a head is ready for harvest. All of the methods were evaluated using ground-truth data from both the UK and Spain, which we also make available to the research community for subsequent algorithm development and result comparison. Cross-validation of the system trained on the UK dataset on the Spanish dataset, and vice versa, indicated good generalization capabilities of the system, confirming the strong potential of low-cost 3D imaging for commercial broccoli harvesting

Agricultural Robotics: The Future of Robotic Agriculture

“…Machine vision approaches offer significant opportunities for enabling autonomy of robotic systems in food production. Vision-based tasks for crop monitoring include phenotyping [29], classifying when individual plants are ready for harvest [30], and quality analysis [31], e.g. detecting the onset of diseases, all with high throughput data.…”

“…Approaches based on analysis of 3D point clouds, e.g. derived from stereo imagery or RGB-D cameras, offer significant promise to achieve robust perception in challenging agricultural environments [30,35].…”

Transfer learning between crop types for semantic segmentation of crops versus weeds in precision agriculture

“…Such vision systems typically include a plant classification component to identify individual crop and weed plants, which is then used as a basis for selecting the required treatment. A number of different vision pipelines, typically consisting of vegetation segmentation followed by classification and based on hand‐crafted features, have been proposed (Bosilj, Duckett, & Cielniak, ; Haug, Michaels, Biber, & Ostermann, ; Hemming & Rath, ; Kusumam, Krajník, Pearson, Duckett, & Cielniak, ; Lottes, Hörferlin, Sander, & Stachniss, ). Conversely, convolutional neural networks (CNNs) can automatically determine complex and highly discriminative features directly from images.…”

Vision techniques for on‐board detection, following, and mapping of moving targets

“…The mapping module integrates the individual transformed detections into a global 3d coordinate frame shared by the UAV team. The map building algorithm used was inspired by a system described in Kusumam, Krajník, Pearson, Duckett, and Cielniak (), which proved reliable operation in field conditions. To associate the object detections coming from the perception system with the objects in the map, we assume that the maximal error of object position estimation is below a specific value, which we denote as .…”