This paper reviews the second challenge on spectral reconstruction from RGB images, i.e., the recovery of wholescene hyperspectral (HS) information from a 3-channel RGB image. As in the previous challenge, two tracks were provided: (i) a "Clean" track where HS images are estimated from noise-free RGBs, the RGB images are themselves calculated numerically using the ground-truth HS images and supplied spectral sensitivity functions (ii) a "Real World" track, simulating capture by an uncalibrated and unknown camera, where the HS images are recovered from noisy JPEG-compressed RGB images. A new, larger-than-ever, natural hyperspectral image data set is presented, containing a total of 510 HS images. The Clean and Real World tracks had 103 and 78 registered participants respectively, with 14 teams competing in the final testing phase. A description of the proposed methods, alongside their challenge scores and an extensive evaluation of top performing methods is also provided. They gauge the state-of-the-art in spectral reconstruction from an RGB image. arXiv:2005.03412v1 [eess.IV] 7 May 2020

In this paper, we describe the Perceptual Image Restoration and Manipulation (PIRM) workshop challenge on spectral image superresolution, motivate its structure and conclude on results obtained by the participants. The challenge is one of the first of its kind, aiming at leveraging modern machine learning techniques to achieve spectral image super-resolution. It comprises of two tracks. The first of these (Track 1) is about example-based single spectral image super-resolution. The second one (Track 2) is on colour-guided spectral image super-resolution. In this manner, Track 1 focuses on the problem of super-resolving the spatial resolution of spectral images given training pairs of low and high spatial resolution spectral images. Track 2, on the other hand, aims to leverage the inherently higher spatial resolution of colour (RGB) cameras and the link between spectral and trichromatic images of the scene. The challenge in both tracks is then to recover a super-resolved image making use of low-resolution imagery at the input. We also elaborate upon the methods used by the participants, summarise the results and discuss their rankings.

Recent advances have shown the great power of deep convolutional neural networks (CNN) to learn the relationship between low and high-resolution image patches. However, these methods only take a single-scale image as input and require large amount of data to train without the risk of overfitting. In this paper, we tackle the problem of multi-modal spectral image super-resolution while constraining ourselves to a small dataset. We propose the use of different modalities to improve the performance of neural networks on the spectral superresolution problem. First, we use multiple downscaled versions of the same image to infer a better high-resolution image for training, we refer to these inputs as a multi-scale modality. Furthermore, color images are usually taken at a higher resolution than spectral images, so we make use of color images as another modality to improve the super-resolution network. By combining both modalities, we build a pipeline that learns to super-resolve using multi-scale spectral inputs guided by a color image. Finally, we validate our method and show that it is economic in terms of parameters and computation time, while still producing state-of-the-art results. 1

We propose a real-time image fusion method using pre-trained neural networks. Our method generates a single image containing features from multiple sources. We first decompose images into a base layer representing large scale intensity variations, and a detail layer containing small scale changes. We use visual saliency to fuse the base layers, and deep feature maps extracted from a pre-trained neural network to fuse the detail layers. We conduct ablation studies to analyze our method's parameters such as decomposition filters, weight construction methods, and network depth and architecture. Then, we validate its effectiveness and speed on thermal, medical, and multi-focus fusion. We also apply it to multiple image inputs such as multiexposure sequences. The experimental results demonstrate that our technique achieves state-of-the-art performance in visual quality, objective assessment, and runtime efficiency.

Developing an augmented reality (AR) system involves multiple algorithms such as image fusion, camera synchronization and calibration, and brightness control, each of them having diverse parameters. This abundance of settings, while allowing for many features, is detrimental to developers as they try to navigate between different combinations and pick the most suitable towards their application. Additionally, the temporally inconsistent nature of the real world makes it hard to build reproducible scenarios for testing and comparison. To help address these issues, we develop a virtual reality (VR) environment that allows simulating a variety of AR configurations 1 . We show the advantages of AR simulation in virtual reality, demonstrate an image fusion AR system and conduct an experiment to compare different fusion methods.