Fast Fourier Color Constancy

Abstract

We present Fast Fourier Color Constancy (FFCC), a color constancy algorithm which solves illuminant estimation by reducing it to a spatial localization task on a torus. By operating in the frequency domain, FFCC produces lower error rates than the previous state-of-the-art by 13 − 20% while being 250 − 3000× faster. This unconventional approach introduces challenges regarding aliasing, directional statistics, and preconditioning, which we address. By producing a complete posterior distribution over illuminants instead of a single illuminant estimate, FFCC enables better training techniques, an effective temporal smoothing technique, and richer methods for error analysis. Our implementation of FFCC runs at ∼ 700 frames per second on a mobile device, allowing it to be used as an accurate, real-time, temporally-coherent automatic white balance algorithm.

Handheld mobile photography in very low light

“…Our "Anisotropic Reproduction Error" (ARE), denoted ∆ a , accounts for the anisotropy of the RGB values in the true image, and downweights errors in missing channels accordingly, resulting in a learned model that produces output images (b) that closely match the ground-truth (a) in these challenging scenes. Below we visualize the error surfaces of traditional metrics ∆ and ∆ r alongside ∆ a using the log-chroma UV histograms of (Barron and Tsai 2017), where luma indicates error (the true illuminant is at the origin and our prediction is a white circle). By conditioning on the mean RGB value of the true image µ t , the ARE's curvature is reduced with respect to red (along the u-axis) but still highly curved with respect to blue and green (along the -axis and diagonally), which accurately reflects performance in this heavily-tinted scene.…”

“…the ARE degrades naturally to the standard reproduction error when the average scene color is gray: In most of the scenes in our dataset, the average true color is close to gray and therefore the error surface of the ARE closely resembles the reproduction error, but in challenging low-light scenes the di erence between the two metrics can be signi cant, as is shown in Figure 12 In addition to its value as a metric for measuring performance, the ARE can be used as an e ective loss during training. To demonstrate this we trained our FFCC model on our dataset using the same procedure as was used in (Barron and Tsai 2017): three-fold cross validation, where hyperparameters are tuned to minimize the "average" error used by that work. We trained four models: a baseline in which we minimize the von Mises negative log-likelihood term used by (Barron and Tsai 2017), and three others in which we replaced the negative log-likelihood with three error metrics: ∆ r , ∆ , and our ∆ a (all of which are di erentiable and therefore straightforward to minimize with gradient descent).…”

“…To demonstrate this we trained our FFCC model on our dataset using the same procedure as was used in (Barron and Tsai 2017): three-fold cross validation, where hyperparameters are tuned to minimize the "average" error used by that work. We trained four models: a baseline in which we minimize the von Mises negative log-likelihood term used by (Barron and Tsai 2017), and three others in which we replaced the negative log-likelihood with three error metrics: ∆ r , ∆ , and our ∆ a (all of which are di erentiable and therefore straightforward to minimize with gradient descent). Results can be seen in Table 3.…”

“…C.2 Additional improvements to FFCC e "Fast Fourier Color Constancy" (FFCC) (Barron and Tsai 2017) algorithm that our white balance solution is built upon was modi ed to improve its speed, quality, and interpretability, as we will describe here. Instead of providing FFCC with a 2-channel histogram of pixel and edge intensities for the entire image, we concatenate this histogram with a 2-channel histogram of pixel and edge intensities for only those pixels that belong to human faces, as predicted by the camera's face detector.…”

AIM 2019 Challenge on Image Demoireing: Methods and Results

“…In the case of digital photography of screens, moire patterns occur when the screen's subpixel layout interferes with the camera's color filter array (CFA). Digital image quality has been improving over the past years, with great improvements introduced in image denoising [25], image demosaicing [6], sharpening [16], automatic white balancing [2], and high dynamic range compression [7]. However, current image signal processing (ISP) pipelines often produce strong moire patterns when presented a photograph of a screen taken with a digital camera.…”