Ranking-Based Salient Object Detection and Depth Prediction for Shallow Depth-of-Field

Abstract

Shallow depth-of-field (DoF), focusing on the region of interest by blurring out the rest of the image, is challenging in computer vision and computational photography. It can be achieved either by adjusting the parameters (e.g., aperture and focal length) of a single-lens reflex camera or computational techniques. In this paper, we investigate the latter one, i.e., explore a computational method to render shallow DoF. The previous methods either rely on portrait segmentation or stereo sensing, which can only be applied to portrait photos and require stereo inputs. To address these issues, we study the problem of rendering shallow DoF from an arbitrary image. In particular, we propose a method that consists of a salient object detection (SOD) module, a monocular depth prediction (MDP) module, and a DoF rendering module. The SOD module determines the focal plane, while the MDP module controls the blur degree. Specifically, we introduce a label-guided ranking loss for both salient object detection and depth prediction. For salient object detection, the label-guided ranking loss comprises two terms: (i) heterogeneous ranking loss that encourages the sampled salient pixels to be different from background pixels; (ii) homogeneous ranking loss penalizes the inconsistency of salient pixels or background pixels. For depth prediction, the label-guided ranking loss mainly relies on multilevel structural information, i.e., from low-level edge maps to high-level object instance masks. In addition, we introduce a SOD and depth-aware blur rendering method to generate shallow DoF images. Comprehensive experiments demonstrate the effectiveness of our proposed method.

Constrained Predictive Filters for Single Image Bokeh Rendering

“…With the development of computer vision, depth estimation [14,30,34], defocus estimation [2,32,7] and saliency detection [54,38,43,56] have made significant progress, which also provide more directions for solving bokeh rendering tasks. Many methods [48,37,11,17] utilize these prior knowledge to synthesize bokeh effects. Luo et al [37] utilize radiance estimation and defocus estimation [47] to predict the relationship between the amount of blur and the intensity of pixels in each image, and then synthesizes the bokeh effect with pre-defined blur kernels.…”

“…Furthermore, Purohit et al also send the estimated saliency maps [15] into the network as input. Xian et al [48] integrate the depth estimation module and saliency detection module into the same network, and synthesis the bokeh effects using a physically motivated method. However, due to the large difference between the dataset used for prior knowledge training and the bokeh dataset, the accuracy of prior knowledge may not be reliable, and the imprecise prior may degrade models.…”

MPIB: An MPI-Based Bokeh Rendering Framework for Realistic Partial Occlusion Effects

“…In the early years, most physically based methods [1,14,31,40,46] entail 3D scene information and are timeconsuming. Recent methods [2,3,7,26,32,33,38,41,44,48], which render bokeh effects from a single image and a corresponding depth map, are more efficient and practical. One classic idea is layered rendering [3,48], i.e., decomposing the scene into multiple layers and independently blurring each layer before compositing them from back to front.…”