An investigation on <scp>worst‐case</scp> spectral reconstruction from <scp>RGB</scp> images via Radiance Mondrian World assumption

Lin, Yi‐Tun; Finlayson, Graham D.

doi:10.1002/col.22843

Cited by 8 publications

(3 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The main ranking protocols of NTIRE competitions also do not account for performance under more difficult imaging conditions (that are still often encountered in the real world). Indeed, more comprehensive benchmarks show that DNNs are generally vulnerable to exposure change [33,34], out-of-scope scenes [30] and scenes without particular image contents [30,54]. In this paper, we will also show that the leading DNN is negatively and significantly affected by image rotation and blur.…”

Section: Related Workmentioning

confidence: 73%

A Rehabilitation of Pixel-Based Spectral Reconstruction from RGB Images

Lin

Finlayson

2023

Sensors

View full text Add to dashboard Cite

Recently, many deep neural networks (DNN) have been proposed to solve the spectral reconstruction (SR) problem: recovering spectra from RGB measurements. Most DNNs seek to learn the relationship between an RGB viewed in a given spatial context and its corresponding spectra. Significantly, it is argued that the same RGB can map to different spectra depending on the context with respect to which it is seen and, more generally, that accounting for spatial context leads to improved SR. However, as it stands, DNN performance is only slightly better than the much simpler pixel-based methods where spatial context is not used. In this paper, we present a new pixel-based algorithm called A++ (an extension of the A+ sparse coding algorithm). In A+, RGBs are clustered, and within each cluster, a designated linear SR map is trained to recover spectra. In A++, we cluster the spectra instead in an attempt to ensure neighboring spectra (i.e., spectra in the same cluster) are recovered by the same SR map. A polynomial regression framework is developed to estimate the spectral neighborhoods given only the RGB values in testing, which in turn determines which mapping should be used to map each testing RGB to its reconstructed spectrum. Compared to the leading DNNs, not only does A++ deliver the best results, it is parameterized by orders of magnitude fewer parameters and has a significantly faster implementation. Moreover, in contradistinction to some DNN methods, A++ uses pixel-based processing, which is robust to image manipulations that alter the spatial context (e.g., blurring and rotations). Our demonstration on the scene relighting application also shows that, while SR methods, in general, provide more accurate relighting results compared to the traditional diagonal matrix correction, A++ provides superior color accuracy and robustness compared to the top DNN methods.

show abstract

Section: Related Workmentioning

confidence: 73%

A Rehabilitation of Pixel-Based Spectral Reconstruction from RGB Images

Lin

Finlayson

2023

Sensors

View full text Add to dashboard Cite

show abstract

“…This was constructed by using a similar approach to that of Ref. [21] where any given patch on the 2d Mondrian grid has a specific width and height, which were randomly drawn from a uniform probability distribution with minimum and maximum width and height limits. The central coordinates of each of the patches were also randomly drawn from a uniform probability distribution and patches generated in this way were superimposed until all pixels on the grid were filled.…”

Section: Mondrian Worldmentioning

confidence: 99%

Mondrian Representation of Real World Scene Statistics

Rowlands,

Finlayson

2023

lim

View full text Add to dashboard Cite

In material appearance we are interested in objectively measuring a physical aspect of a material, such as reflectance, and also in understanding how we see that material. In perceptual experiments we typically display simple stimuli to an observer, record their response, and then try to build a theory of why the observer responded in a given way. Often the latter model is implemented as a computer algorithm, and many of these are, for example, now implemented in camera pipelines for smartphones. However, the stimuli that are shown to observers are necessarily either very simple, such as rectangular patches of colour, or small in number. This raises the question as to whether the recorded responses to simple stimuli actually shed light on how we perceive scenes in the real world.In this paper, we look at a specific example of perceptual stimuli: Mondrian images, and investigate the extent to which, in the sense of their autocorrelation matrix, they represent the real world. We show that by modelling paths of pixels through an image using a statistical model that captures the statistics of real Mondrians, the autocorrelation matrix of Mondrians is Toeplitz, and moreover this Toeplitz structure is also found in real images. Although Mondrian images do not contain typical visual cues, our path model can be tuned to replicate the statistics of real images in the autocorrelation sense. The practical utility of this method is that paths through images and their autocorrelation statistics are a key tool for developing algorithms to predict the perceptual response to complex scenes. For example, this approach is at the foundation of retinex image processing. Experiments validate our method.

show abstract

“…Spectral datasets, on the other hand, are often limited to patch-based (e.g. "Mondrian-like [1]") content, which for example limits the applicability of advanced methods for spectral reconstruction that rely on the analysis of scene content.…”

Section: Introductionmentioning

confidence: 99%

A General-purpose Pipeline for Realistic Synthetic Multispectral Image Dataset Generation

Buzzelli,

Tchobanou,

Schettini

et al. 2023

cic

View full text Add to dashboard Cite

A pipeline for the generation of synthetic dataset of spectral scenes, with corresponding sensor readings, is here proposed. The pipeline is composed of two main parts: Part 1: Image pixel reflectance assignment. Individual pixels from an input sRGB image dataset are replaced with appropriate reflectance spectra from a given non-image reflectance dataset. The resulting dataset of reflectance images is considered the starting point for simulated sensor acquisition. Part 2: Simulated sensor acquisition. Each spectral reflectance image in the dataset is illuminated with an illuminant spectra to produce a radiance image. The resulting dataset of radiance images is then synthetically read from the simulated sensors (camera and ambient multispectral sensor) of the Huawei P50 phone, using the corresponding sensors transmittance information. The capability of generating any large-scale, diverse, and annotated synthetic spectral datasets can facilitate the development of data-driven imaging algorithms, and foster reproducible research.

show abstract

An investigation on worst‐case spectral reconstruction from RGB images via Radiance Mondrian World assumption

Cited by 8 publications

References 44 publications

A Rehabilitation of Pixel-Based Spectral Reconstruction from RGB Images

A Rehabilitation of Pixel-Based Spectral Reconstruction from RGB Images

Mondrian Representation of Real World Scene Statistics

A General-purpose Pipeline for Realistic Synthetic Multispectral Image Dataset Generation

Contact Info

Product

Resources

About