Separating Reflection and Transmission Images in the Wild

Abstract

The reflections caused by common semi-reflectors, such as glass windows, can impact the performance of computer vision algorithms. State-of-the-art methods can remove reflections on synthetic data and in controlled scenarios. However, they are based on strong assumptions and do not generalize well to real-world images. Contrary to a common misconception, real-world images are challenging even when polarization information is used. We present a deep learning approach to separate the reflected and the transmitted components of the recorded irradiance, which explicitly uses the polarization properties of light. To train it, we introduce an accurate synthetic data generation pipeline, which simulates realistic reflections, including those generated by curved and non-ideal surfaces, non-static scenes, and high-dynamic-range scenes.

Reflection Separation via Multi-bounce Polarization State Tracing

“…Wan et al [17] (Wen et al) apply a multi-scale strategy on the learning network to improve the target details. We also compared our results with polarization guided reflection separation methods [23,24](ReflectNet and Lyu et al), which are proposed most recently. They assume a simplified model with only one transmitted light path, whereas multiple reflections between the two glass surfaces are ignored.…”

“…By estimating the motion between the transmitted and reflected images with different strategies [20][21][22], researchers manage to separate them for reflection removal. Recently, there is an emerging interest in polarization guided image reflection separation [23][24][25]. With multiple images captured for the same scene at different polarization angles, the reflections from glasses are separated by applying independent components analysis [26].…”

“…However, they failed to consider the thickness of the transparent medium, which introduced ghost images. By leveraging the properties of polarized light, the reflection and transmission images are separated with more impressive results in recent approaches [24]. Lyu et al [23] takes unpolarized and polarized images for reflection separation.…”

A Comprehensive Introduction to Photometric 3D-Reconstruction

“…In addition to the use of polarisation information for 3D-reconstruction, recently several other applications are using polarisation for different tasks. For example, for image segmentation [104], robot dynamic navigation [13,14], image enhancement [99,100], and reflection separation by a deep learning approach [66], which simplifies previous works requiring three images from different polariser angles [59,101,124]. For more details on possible applications, we refer the interested reader to Chapter 6 of this volume.…”

The Mertens Unrolled Network (MU-Net): A high dynamic range fusion neural network for through the windshield driver recognition

“…In glass artifact suppression, algorithms look to effectively remove artifacts such as reflection, refraction, and glare, that are introduced by glass into real-world images. [13][14][15][16] While these works are useful within their own context, they are solutions designed around separate problems and therefore sub-optimal to solve the through-windshield recognition task, due to less-stringent time or accuracy constraints on their output Convolutional Neural Networks (CNNs) are the the current SOTA approaches for a large number of image processing tasks. Some recent successes of the neural networks can be attributed to their ability learn and optimize the behavior previously established deterministic algorithms, as is pointed out in.…”