Computer vision‐based tree trunk and branch identification and shaking points detection in Dense‐Foliage canopy for automated harvesting of apples

Abstract

Fresh market apples are one of the high-value crops in the United States.Washington alone has produced two-thirds of the annual national production in the past 10 years. However, the availability of seasonal labor is increasingly uncertain. Shake-and-catch automated harvesting solutions have, therefore, become attractive for addressing this challenge. One of the significant challenges in applying this harvesting system is effectively positioning the end-effector at appropriate excitation locations. A computer vision system was used for automatically identifying appropriate locations. Convolutional neural networks (CNNs) were utilized to identify the tree trunks and branches for supporting the automated excitation locations determination. Three CNN architectures were employed: Deeplab v3+ ResNet-18, VGG-16, and VGG-19. Four pixel classes were predefined as branches, trunks, apples, and leaves to segment the canopies trained to simple, narrow, accessible, and productive tree architectures with varying foliage density. Results on Fuji cultivar showed that ResNet-18 outperformed the VGGs in identifying branches and trunks based on all three evaluation measures: per-class accuracy (PcA), intersection over union (IoU), and boundary-F1 score (BFScore). ResNet-18 achieved a PcA of 97%, IoU of 0.69, and BFScore of 0.89. The ResNet-18 was further evaluated for its robustness with other test canopy images. When applied this method to one of the highest density cultivars of Scifresh, results showed it can achieve IoUs of 0.41 and 0.62 and BFScores of 0.71 and 0.86 for branches and trunks. Such identification result helped to get a 72% of auto-determined shaking points being the "good" category identified by human experts.

Understanding the dynamic properties of trees using the motions constructed from multi-beam flash light detection and ranging measurements

“…Figure 7 shows the modal frequencies ( f 1 , f 2 ), modal damping ratios ( ξ 1, ξ 2 ) and mode shapes (ϕ1,ϕ2) of the tree analysed using SSI (equations (2.8)–(2.10)). System orders N ranging from 4 to 30 were tested [47], and the first five vibration cycles were adopted to construct the Hankel matrix H (equation (2.5)). In this study, two vibration modes can be consistently extracted from SSI in all the tests.…”

“…Note that increasing the order of polynomial function normally improves the fitting accuracy. However, due to the overfitting problem, an excessively high order may not necessarily enhance the results of using the polynomial function to describe the geometry of branches [47]. Recalling that the vibration of trees may behave as cantilever beams [10], the associated vibration shapes could be approximated by a third-order polynomial function.…”

De-Handing Technologies for Banana Postharvest Operations—Updates and Challenges

“…A major advantage of developing machine vision systems is the ability to provide a reliable and efficient automation solution for the banana de-handing operation in a fast and precise manner. As a promising technology to accurately perceive the environment for automated/robotic operation, machine vision has been widely investigated for various applications in fruit crops including phenotyping and canopy detection [19][20][21], branch and trunk identification [22][23][24][25] and fruit localization [26,27]. Different from the industrial environment, fruit orchards are complex, uncertain, variable and with many uncontrollable factors such as light and wind.…”

Optical flow-based branch segmentation for complex orchard environments

“…Amatya and Karkee [5] train a Bayesian classifier to segment and reconstruct branches on images of sweet cherry trees taken at night, while [6] uses an encoderdecoder network to segment out branches, wires, and fruit in a kiwifruit orchard. Depth information is a common modality for filtering out unwanted background noise: [7], [8] use depth information to filter out all points beyond a specified threshold before feeding the RGB image through a neural network, while [9], [10] feed the depth channel directly into the neural network to filter out background noise. Yang et al [11] produce a mask with a raw RGB image but postprocess the mask with depth data to create an accurate tree model for localization.…”