A Feature Integrated Saliency Estimation Model for Omnidirectional Immersive Images

Abstract

Omnidirectional, or 360°, cameras are able to capture the surrounding space, thus providing an immersive experience when the acquired data is viewed using head mounted displays. Such an immersive experience inherently generates an illusion of being in a virtual environment. The popularity of 360 • media has been growing in recent years. However, due to the large amount of data, processing and transmission pose several challenges. To this aim, efforts are being devoted to the identification of regions that can be used for compressing 360 • images while guaranteeing the immersive feeling. In this contribution, we present a saliency estimation model that considers the spherical properties of the images. The proposed approach first divides the 360 • image into multiple patches that replicate the positions (viewports) looked at by a subject while viewing a 360 • image using a head mounted display. Next, a set of low-level features able to depict various properties of an image scene is extracted from each patch. The extracted features are combined to estimate the 360 • saliency map. Finally, bias induced during image exploration and illumination variation is fine-tuned for estimating the final saliency map. The proposed method is evaluated using a benchmark 360 • image dataset and is compared with two baselines and eight state-of-the-art approaches for saliency estimation. The obtained results show that the proposed model outperforms existing saliency estimation models.2 of 15 approaches towards the saliency estimation rely on the detection of image components attracting human attention, i.e., color, intensity, and texture [6,7].Other approaches exploit Gestalt's psychological studies [8,9], according to which human perception focuses on figures more than on background elements. In [10] a Boolean map based saliency model (BMS) for 2D images is presented and an extended version of this approach, the extended Boolean map saliency approach (EBMS), is proposed in [11].Both BMS and EBMS do not consider the geometry-related features of an image and cannot directly be used for omnidirectional content since they do not address the problem of spherical projection of 360 • images that may cause artifacts. Fang et al. [12] adapt the traditional 2D saliency approach to 360 • images. Other methods adopt low-level features [12,13] or a combination of low and high-level features [14][15][16] for 360 • image saliency estimation.Recently, methods have been proposed to take into account the artifacts caused by the spherical projections. In [17], performances of three saliency estimation methods: Graph-based visual saliency, ensemble of deep networks (eDN), and the saliency attentive model (SAM) are compared. eDN exploits a six-multilayer structure to identify the salient regions through hyper-parameter optimisation. The ResNet architecture combined with pre-trained VGG-16 is used in the SAM model. Each model is tested using three types of 360 • image projection formats: Continuity, cube, and combined equirectangular image projection.Low...

Delivery of omnidirectional video using saliency prediction and optimal bitrate allocation

“…For both VGG-TL and VGG-MLNet, our framework first need to create various overlapping 2D projections of scene using spherical coordinate at fixed image resolutions (i.e. 512×512 pixels and 480×640 pixels for VGG-TL and VGG-MLNet models respectively) with θ F oV = 120°at θ viewport−center = [0, 15 , ..., 330, 345] and φ viewport−center = [-90, -75, ..., 0, ..., 75, 90] similar to various works on 360°data processing [13][14][15]31]. Then, each 2D image patch from the scene is used in VGG-TL or VGG-MLNet to obtain saliency maps of each extracted view-ports.…”

No-Reference Light Field Image Quality Assessment Exploiting Saliency

“…where the normalized image or one of the five saliency map models (Itti [28], GBVS [70], Geometry [71], BMS [72], and EBMS [73]) are computed. The image pairs are obtained as follows:…”

JPEG Lifting Algorithm Based on Adaptive Block Compressed Sensing

“…at is, the block compression measurement matrix could be used as the forward discrete cosine transform (FDCT) matrix in the JPEG coding, and the inverse discrete cosine transform (IDCT) process is replaced by sparse reconstruction. In addition, multiple feature saliency and noise analysis are introduced to implement adaptive control of the observation matrix and minimal error iterative reconstruction [8,9].…”