End-To-End Projector Photometric Compensation

Abstract

Projector photometric compensation aims to modify a projector input image such that it can compensate for disturbance from the appearance of projection surface. In this paper, for the first time, we formulate the compensation problem as an end-to-end learning problem and propose a convolutional neural network, named CompenNet, to implicitly learn the complex compensation function. Compen-Net consists of a UNet-like backbone network and an autoencoder subnet. Such architecture encourages rich multilevel interactions between the camera-captured projection surface image and the input image, and thus captures both photometric and environment information of the projection surface. In addition, the visual details and interaction information are carried to deeper layers along the multi-level skip convolution layers. The architecture is of particular importance for the projector compensation task, for which only a small training dataset is allowed in practice.Another contribution we make is a novel evaluation benchmark, which is independent of system setup and thus quantitatively verifiable. Such benchmark is not previously available, to our best knowledge, due to the fact that conventional evaluation requests the hardware system to actually project the final results. Our key idea, motivated from our end-to-end problem formulation, is to use a reasonable surrogate to avoid such projection process so as to be setup-independent. Our method is evaluated carefully on the benchmark, and the results show that our end-to-end learning solution outperforms state-of-the-arts both qualitatively and quantitatively by a significant margin.

CompenNet++: End-to-End Full Projector Compensation

“…This paper builds upon preliminary conference papers CompenNet [20] and CompenNet++ [19] and significantly extends them in various aspects. 1We redesigned the photometric compensation subnet as removing the surface from the projector input image in the feature space, based on which we propose a novel photometric compensation subnet named CompenNeSt (i.e., the photometric part of CompenNeSt++, the additional "S" stands for siamese).…”

“…These requirements often create bottlenecks for generalization of projector systems. Full projector geometric correction and photometric compensation [2], [4], [17], [44], [51], [52] aims to address this issue by modifying a projector input image to compensate for the projection setup geometry [5], [37], [42], [43], [54] and associated photometric environment [1], [3], [15], [20], [60]. In the rest of the text, we call it full compensation for conciseness.…”

“…1We redesigned the photometric compensation subnet as removing the surface from the projector input image in the feature space, based on which we propose a novel photometric compensation subnet named CompenNeSt (i.e., the photometric part of CompenNeSt++, the additional "S" stands for siamese). Compared with our previous CompenNet [20] (i.e., the photometric part of CompenNet++ [19]), CompenNeSt is designed with a siamese encoder to explicitly apply the same feature transformation to the surface image and the projector input image, using which we can perform photometric compensation by subtracting the surface pattern from the projector input image in the feature space. Moreover, minor tweaks on skip and transposed convolutional layers are shown to further improve model performance.…”

“…Such studies were not available in our previous papers. 3We propose a novel pre-training strategy using Blender [6] rendered synthetic dataset, which greatly improves the practicability compared with the naive pre-training method of Compen-Net/CompenNet++ [19], [20] and it can significantly reduce the number of training images to 8 and training time to 3 minutes. For the benefit of the society, the source code, evaluation benchmark and experimental results are publicly available at https://github.com/BingyaoHuang/ CompenNeSt-plusplus.…”

Illuminated Focus: Vision Augmentation using Spatial Defocusing via Focal Sweep Eyeglasses and High-Speed Projector

“…On the other hand, we can consider synthetic focal modulation, where the appearance of a blurred real scene is synthesized by convolution computation, as in VST-AR systems, and is reproduced by superimposed projected imagery. Researchers have developed various radiometric compensation techniques that enable pixel-wise color manipulation of a textured real surface [10,18], which could be used to reproduce the blurred appearance. However, these techniques cannot make a real surface appear completely blurred due to the limited dynamic range of current projectors [3].…”