Attempting to Estimate the Unseen—Correction for Occluded Fruit in Tree Fruit Load Estimation by Machine Vision with Deep Learning

Abstract

Machine vision from ground vehicles is being used for estimation of fruit load on trees, but a correction is required for occlusion by foliage or other fruits. This requires a manually estimated factor (the reference method). It was hypothesised that canopy images could hold information related to the number of occluded fruits. Several image features, such as the proportion of fruit that were partly occluded, were used in training Random forest and multi-layered perceptron (MLP) models for estimation of a correction factor per tree. In another approach, deep learning convolutional neural networks (CNNs) were directly trained against harvest count of fruit per tree. A R2 of 0.98 (n = 98 trees) was achieved for the correlation of fruit count predicted by a Random forest model and the ground truth fruit count, compared to a R2 of 0.68 for the reference method. Error on prediction of whole orchard (880 trees) fruit load compared to packhouse count was 1.6% for the MLP model and 13.6% for the reference method. However, the performance of these models on data of another season was at best equivalent and generally poorer than the reference method. This result indicates that training on one season of data was insufficient for the development of a robust model.

Technologies for Forecasting Tree Fruit Load and Harvest Timing—From Ground, Sky and Time

“…For example, Sarron et al [9] described the use of a human expert to categorize fruit load density to one of four classes as one input to a model for fruit load estimation. Koirala et al [12] described a machine vision-based classification of canopy images to three crop load densities using a 'Xception_classification' model and a Silhouette score as a first step in a pipeline for a machine vision-based estimation of tree fruit load (see later sections).…”

“…For the prediction of fruit load in a different season to that used for model training, the RMSE and R 2 between estimated and harvested apple yield were 2.6 kg/tree and 0.62 for early fruit growth (small, green fruit) and 2.5 kg/tree and 0.75 near harvest (red, large fruit), for trees with an average 18 kg of fruit. In a later attempt in the direct prediction of total tree fruit load, deep learning techniques were employed [12], and good predictions of current season tree fruit loads were achieved, but predictions of fruit load per tree for a subsequent season were poor. Further work should be undertaken to progress this concept.…”

Estimation of Fruit Load in Australian Mango Orchards Using Machine Vision

“…Use of the multi-view machine vision estimate There remains a need to develop better tools for estimation of an occlusion factor for a given orchard. Koirala et al [16] unsuccessfully attempted estimation of a per tree occlusion factor based on machine vision features such as the proportion of partly occluded to non-occluded fruit. A random forest model on the ratio of machine vision to packhouse count using inputs of various orchard attributes explained only 39% of the variance in the ratio, with 10, 8, 6, 4 and 3% attributed to tree age, SD of tree crown area, mean of tree crown area row spacing and tree spacing, respectively (Appendix B).…”

“…The sum of the manual fruit count for 18 trees in each orchard was regressed to the MangoYOLO pipeline count of fruit in the two images for these trees to estimate an 'occlusion factor' for each orchard. This correction factor was then applied to the machine vision count of all other trees, following the approach of Koirala et al [16].…”

Real UAV-Bird Image Classification Using CNN with a Synthetic Dataset

“…In cases where the number of samples is limited and a CNN-based machine learning algorithm is used, the success of the network is low. To increase the performance of the network, many samples are needed [35]. To increase the number of samples, synthetic images can be generated by using 3D models and game engines.…”