Training generative adversarial networks for optical property mapping using synthetic image data

Abstract

We demonstrate the training of a generative adversarial network (GAN) for the prediction of optical property maps (scattering and absorption) using spatial frequency domain imaging (SFDI) image data sets that are generated synthetically with a free open-source 3D modelling and rendering software, Blender. The flexibility of Blender is exploited to simulate 5 models with real-life relevance to clinical SFDI of diseased tissue: flat samples containing a single material, flat samples containing 2 materials, flat samples containing 3 materials, flat samples with spheroidal tumours and cylindrical samples with spheroidal tumours. The last case is particularly relevant as it represents wide-field imaging inside a tubular organ e.g. the gastro-intestinal tract. In all 5 scenarios we show the GAN provides an accurate reconstruction of the optical properties from single SFDI images with a mean normalised error ranging from 1.0-1.2% for absorption and 1.1%-1.2% for scattering, resulting in visually improved contrast for tumour spheroid structures. This compares favourably with the ∼10% absorption error and ∼10% scattering error achieved using GANs on experimental SFDI data. Next, we perform a bi-directional cross-validation of our synthetically-trained GAN, retrained with 90% synthetic and 10% experimental data to encourage domain transfer, with a GAN trained fully on experimental data and observe visually accurate results with an error of 6.3%-10.3% for absorption and 6.6%-11.9% for scattering. Our synthetically trained GAN is therefore highly relevant to real experimental samples but provides the significant added benefits of large training datasets, perfect ground-truths and the ability to test realistic imaging geometries, e.g. inside cylinders, for which no conventional single-shot demodulation algorithms exist. In the future, we expect that the application of techniques such as domain adaptation or training on hybrid real-synthetic datasets will create a powerful tool for fast, accurate production of optical property maps for real clinical imaging systems.

Ultra-miniature dual-wavelength spatial frequency domain imaging for micro-endoscopy

“…This causes both noise and some inaccuracies in optical properties. The noise has been partially addressed by varying the spatial frequency but could be further improved by normalizing for laser power, using AI 45 for noise-reduction reconstruction or building custom LUTs based on non-ideal projection patterns 32 . The underestimation of absorption at larger absorption levels can be up to 60%, though this too may be reduced by alternative calibration procedures 39 .…”

Reflection-mode transmission matrix reconstruction using neural networks in optical fiber imaging systems

“…Also, this model learns the dynamic representations of the fibre TMs, which in practice, allows fibre perturbations in a real-world system. Further considerations on the demanding requirement of computational resources is required with large sizes of TM, where Generative adversarial network (GAN) 15 can be employed to generate additional data and some matrices compression operators (e.g. factorization 16 , decomposition 17 etc.)…”

Designing and simulating realistic spatial frequency domain imaging systems using open-source 3D rendering software

“…Another potential application of this system could be to generate large SFDI data sets that may be used in lieu of or in addition to experimental data. Such data sets could be used to improve optical property uncertainty measurements by creating large look up tables for specific system setups [ 54 ] or to train deep-learning SFDI recovery systems [ 26 , 55 ].…”

“…Cycles simulates volume scattering inside objects using a Henyey-Greenstein Phase function, which is commonly also used in Monte-Carlo simulations of tissue [ 21 , 22 ]. Blender has previously been used for three-dimensional shape measurement of additive manufacturing parts with complex geometries [ 23 ], for the development of anatomically accurate meshes to use in Monte Carlo light simulations [ 24 ], and for the generation of SFDI image data sets to train neural networks [ 25 , 26 ]. By using Blender for both geometry specification (i.e.…”