A Data-Centric Augmentation Approach for Disturbed Sensor Image Segmentation

Abstract

Deep learning methods have become increasingly popular for optical sensor image analysis. They are adaptable to specific tasks and simultaneously demonstrate a high degree of generalization capability. However, applying deep neural networks to problems with low availability of labeled training data can lead to a model being incapable of generalizing to possible scenarios that may occur in test data, especially with the occurrence of dominant imaging artifacts. We propose a data-centric augmentation approach based on generative adversarial networks that overlays the existing labeled data with synthetic artifacts that are generated from data not present in the training set. This augmentation leads to a more robust generalization capability in semantic segmentation. Our method does not need any additional labeling and does not lead to additional memory or time consumption during inference. Further, we find it to be more effective than comparable approaches that are based on procedurally generated disturbances and the direct use of real disturbances. Building upon the improved segmentation results, we observe that our approach leads to improvements of 22% in the F1-score for an evaluated detection problem, which promises significant robustness towards future disturbances. In the context of sensor-based data analysis, the compensation of image artifacts is a challenge. When the structures of interest are not clearly visible in an image, algorithms that can cope with artifacts are crucial for obtaining the desired information. Thereby, the high variation of artifacts, the combination of different types of artifacts, and their similarity to signals of interest are specific issues that have to be considered in the analysis. Despite the high generalization capability of deep learning-based approaches, their recent success was driven by the availability of large amounts of labeled data. Therefore, the provision of comprehensive labeled image data with different characteristics of image artifacts is of importance. At the same time, applying deep neural networks to problems with low availability of labeled data remains a challenge. This work presents a data-centric augmentation approach based on generative adversarial networks that augments the existing labeled data with synthetic artifacts generated from data not present in the training set. In our experiments, this augmentation leads to a more robust generalization in segmentation. Our method does not need additional labeling and does not lead to additional memory or time consumption during inference. Further, we find it to be more effective than comparable augmentations based on procedurally generated artifacts and the direct use of real artifacts. Building upon the improved segmentation results, we observe that our approach leads to improvements of 22% in the F1-score for an evaluated detection problem. Having achieved these results with an example sensor, we expect increased robustness against artifacts in future applications.

The Employment of the Surface Plasmon Resonance (SPR) Microscopy Sensor for the Detection of Individual Extracellular Vesicles and Non-Biological Nanoparticles

“…Thus, different methods for analyzing recorded images have been developed and described [ 22 ]. Recent improvements are based on machine learning methods and exploit their adaptivity to learning a tailored analysis from training data in order to find the characteristic features of particle regions [ 27 , 28 , 29 , 30 ]. Although individual approaches differ, they can usually be assigned to a meta-pipeline, as illustrated in Figure 7 .…”

“…An example of a concrete implementation is an approach that uses a U-Net [ 31 ] on preprocessed images to generate features and segment them in one module [ 27 ]. The candidate proposal is achieved via a Difference-of-Gaussian-based detector [ 32 ], which processes the output of the U-Net.…”

“…Next, the candidates of all the images are connected over time and filtered based on their length and missing frames in between multiple detections at one image region. As optical disturbances appear in each recording and can also be seen in different types—e.g., random noise, fixed-pattern noise, wave-like, or line-like artifacts [ 27 ]—we developed an approach for achieving high robustness against image artifacts [ 27 ]. In this approach, a U-Net was trained on real examples showing particles of interest combined with disturbance patterns.…”

“…These patterns were generated by a generative adversarial network (GAN) [ 33 ] that was trained on real disturbance patterns and was used to create an arbitrary amount of synthetic but realistic-looking images containing artifacts. By this method, the robustness of the detection against imaging artifacts increased [ 27 ]. In the following, we also use this analysis method as a metric for optimizing the settings of the motors for reliable detection of particles.…”

“…In order to detect particle candidate regions in a preprocessed image, we used a previously developed approach based on a 5-layer U-Net [ 31 ] architecture, with 8 filters in the first layer. The way it was trained, as described in Section 2.3.2 , optimized it to be robust for imaging artifacts: in this way, false detections are significantly reduced [ 27 ] so that the detection of the focus region becomes more reliable.…”