GAN meets chemometrics: Segmenting spectral images with pixel2pixel image translation with conditional generative adversarial networks

Abstract

In analytical chemistry, spectral imaging of complex analytical systems is commonly performed. A major task in spectral imaging analysis is to extract signals related to important analytes present in the imaged scene. Hence, the first task of spectral image analysis is to perform an image segmentation to extract the relevant signals to analyze. However, in the chemometric domain, the traditional image segmentation methods are limited either to threshold-based or pixel-wise classification, therefore, no approache uses the contextual information present in the imaging scene. This study presents for the first time a pixel2pixel (p2p) image translation using conditional generative adversarial networks (cGAN) for the segmentation of spectral images. The p2p cGAN trains two neural models simultaneously where one model (generator) learns to segment and the other model learns to detect if the segmentation performed by the generator model is correct. During the process of generation and detection, the model automatically learns to segment the spectral images accurately. As an application of the p2p cGAN, a case of segmenting visible and near-infrared spectral images of plants was presented. Furthermore, as a comparison, threshold-based and pixel-wise image classification based on partial least-square discriminant analysis were also presented. The results showed that the p2p cGAN based image translation performed the best segmentation task with an intersection over union score of 0.95 AE 0.04. The advanced new DL based image processing approaches can complement spectral image processing.

Generative data augmentation and automated optimization of convolutional neural networks for process monitoring

“…While other methods solely take variations in the spectral domain into account, e.g., by modifying the spectra based on variations in model coefficients ( Bjerrum et al, 2017 ; Blazhko et al, 2021 ), the LSA method projects the information to a latent space in the spectral domain and also accounts for variations in the concentration domain. This idea is similar to algorithms employed in variational autoencoding or GANs ( Guo et al, 2020 ; Mishra and Herrmann, 2021 ), while maintaining interpretability by employing simple numerical methods. By using a cross-validation-based reconstruction approach, we showed that the LSA method can be tuned by various hyperparameters and is able to reconstruct the hold-out test set of the experimental data with a maximum error of roughly 5% and 15% for data sets 1 and 2, respectively.…”

“…Both mentioned approaches solely address the variations in the spectral domain and do not extract component-specific information for augmenting experimental data. Other ML approaches have been tested using generative adversarial networks (GANs) ( Wu et al, 2021 ; Mishra and Herrmann, 2021 ; G. McHardy et al, 2023 ) or variational autoencoders (VAEs) ( Guo et al, 2020 ), where the different input data are projected onto so-called latent structures before they are recombined into in silico representations. Both GANs and VAEs involve neural network structures and hence increase the overall complexity of the approach due to additional hyperparameters.…”

SAR2NDVI: a pix2pix generative adversarial network for reconstructing field-level normalized difference vegetation index time series using Sentinel-1 synthetic aperture radar data

“…The rise in computing power and data abundance has led to the adoption of deep learning across various domains. Particularly, GANs are gaining popularity in various domains 26 – 29 However, the potential of GANs in the earth observation domain is relatively underexplored.…”

“…Particularly, GANs are gaining popularity in various domains. [26][27][28][29] However, the potential of GANs in the earth observation domain is relatively underexplored. Some of the recent GAN-based research works are discussed in this section.…”

Image segmentation with boundary-to-pixel direction and magnitude based on watershed and attention mechanism

“…With further development, novel segmentation approaches, such as DeepLab [17] based on altrous convolutions was proposed to handle the problem of segmenting objects at multiple scales. In order to make adequate use of the semantic context information of image scene, [18,19] proposed a conditional generation adversarial network (cGAN) to solve the general pixel-to-pixel mapping problem, and automatically learned to segment the image accurately. [20] trained the network end-to-end, pixel-to-pixel on semantic segmentation to reduce parameter redundancy and time cost.…”