Tree Trunk Recognition in Orchard Autonomous Operations under Different Light Conditions Using a Thermal Camera and Faster R-CNN

Abstract

In an orchard automation process, a current challenge is to recognize natural landmarks and tree trunks to localize intelligent robots. To overcome low-light conditions and global navigation satellite system (GNSS) signal interruptions under a dense canopy, a thermal camera may be used to recognize tree trunks using a deep learning system. Therefore, the objective of this study was to use a thermal camera to detect tree trunks at different times of the day under low-light conditions using deep learning to allow robots to navigate. Thermal images were collected from the dense canopies of two types of orchards (conventional and joint training systems) under high-light (12–2 PM), low-light (5–6 PM), and no-light (7–8 PM) conditions in August and September 2021 (summertime) in Japan. The detection accuracy for a tree trunk was confirmed by the thermal camera, which observed an average error of 0.16 m for 5 m, 0.24 m for 15 m, and 0.3 m for 20 m distances under high-, low-, and no-light conditions, respectively, in different orientations of the thermal camera. Thermal imagery datasets were augmented to train, validate, and test using the Faster R-CNN deep learning model to detect tree trunks. A total of 12,876 images were used to train the model, 2318 images were used to validate the training process, and 1288 images were used to test the model. The mAP of the model was 0.8529 for validation and 0.8378 for the testing process. The average object detection time was 83 ms for images and 90 ms for videos with the thermal camera set at 11 FPS. The model was compared with the YOLO v3 with same number of datasets and training conditions. In the comparisons, Faster R-CNN achieved a higher accuracy than YOLO v3 in tree truck detection using the thermal camera. Therefore, the results showed that Faster R-CNN can be used to recognize objects using thermal images to enable robot navigation in orchards under different lighting conditions.

Navigation of an Autonomous Spraying Robot for Orchard Operations Using LiDAR for Tree Trunk Detection

“…To facilitate such an environment, research was conducted to develop such a high-density orchard, and observing the navigation with only LiDAR is also possible in some limited conditions. However, to achieve more accurate navigation results, other visual sensors can be used in combination with LiDAR, such as thermal cameras, which also have high potential under low-light conditions for tree detection, as reported in our previous research [ 32 ]. To avoid weeds and external effects, deep learning is also reported with training and testing datasets of tree trunk for positioning reference.…”

Detection of Tip-Burn Stress on Lettuce Grown in an Indoor Environment Using Deep Learning Algorithms

“…Additionally, as mentioned before, it is very difficult for even a human to see and detect the tip-burn under these light conditions. In this case, a thermal camera may be useful for collecting a dataset under different light conditions, as it is not affected by visible light [ 42 ]. Apart from that, there may also be some errors and inconsistencies during the labeling process with certain tip-burn locations not being labeled properly.…”

A Perspective on AI-Based Image Analysis and Utilization Technologies in Building Engineering: Recent Developments and New Directions

“…As in the case of photomodeling, the use of AI technologies can lead to greater precision of analysis in non-ideal conditions such as those of variable lighting conditions [22]. Furthermore, if one thinks of the very large scale, a whole series of very interesting studies in the field of biology already exist [23] and whose principles can easily be applied to the surveillance and creation of databases at an urban and structural level [24].…”