Registration and Fusion of Close-Range Multimodal Wheat Images in Field Conditions

Abstract

Multimodal images fusion has the potential to enrich the information gathered by multi-sensor plant phenotyping platforms. Fusion of images from multiple sources is, however, hampered by the technical lock of image registration. The aim of this paper is to provide a solution to the registration and fusion of multimodal wheat images in field conditions and at close range. Eight registration methods were tested on nadir wheat images acquired by a pair of red, green and blue (RGB) cameras, a thermal camera and a multispectral camera array. The most accurate method, relying on a local transformation, aligned the images with an average error of 2 mm but was not reliable for thermal images. More generally, the suggested registration method and the preprocesses necessary before fusion (plant mask erosion, pixel intensity averaging) would depend on the application. As a consequence, the main output of this study was to identify four registration-fusion strategies: (i) the REAL-TIME strategy solely based on the cameras’ positions, (ii) the FAST strategy suitable for all types of images tested, (iii) and (iv) the ACCURATE and HIGHLY ACCURATE strategies handling local distortion but unable to deal with images of very different natures. These suggestions are, however, limited to the methods compared in this study. Further research should investigate how recent cutting-edge registration methods would perform on the specific case of wheat canopy.

Wheat Ear Segmentation Based on a Multisensor System and Superpixel Classification

“…Secondly, image preprocessing could also lead to segmentation errors; raw images were in a first step prompted to image registration. [21] mentioned some imperfections during this preprocessing. Even though an erosion operation and a blur filter were applied, weird organ deformations were sometimes observed in windy situations (Figure 8(a)).…”

“…Image Registration. The images from the multispectral camera array and the RGB images were registered using a B-spline-based method [21,22] (Figure 1). After this operation, the images could be aligned pixel to pixel to form a single multichannel image containing the multispectral and RGB information at the pixel level.…”

Global Wheat Head Detection 2021: An Improved Dataset for Benchmarking Wheat Head Detection Methods

“…This was the case for Arvalis (Arvalis_7 to Arvalis_12), University of Queensland (UQ_7 to UQ_11), Nanjing Agricultural University (NAU_2 and NAU_3) and University of Kyoto (Ukyoto_1). In addition, 14 new sub-datasets were included, coming from 5 new countries: Norway (NMBU), Belgium (Université of Liège [23]), United States of America (Kansas State University [24], TERRA-REF [7]), Mexico (CIMMYT), and Republic of Sudan (Agricultural Research Council). All these images were acquired at a ground sampling distance between 0.2 and 0.4mm, i.e., similar to that of the images in the GWHD_2020.…”

Illumination-aware image fusion for around-the-clock human detection in adverse environments from Unmanned Aerial Vehicle

“…Before Image fusion, the camera should be calibrated to provide intrinsic camera parameters and distortion coefficients to correct image distortion (Dandrifosse et al, 2021). OpenCV library was used to calibrate the camera.…”

“…A good optical and thermal image fusion method should be able to keep the thermal radiation information in thermal images and the texture detail information in optical images (Piao et al, 2019). The registration of optical and thermal images is a vital preliminary step for image fusion, object detection and tracking, and remote sensing to eliminate the offset between images (Yu et al, 2019;Ding et al, 2021;Dandrifosse et al, 2021). Although many studies exist for optical and thermal image registration, studies for UAV-based platforms are still rare (Meng et al, 2021).…”