Deep360Up: A Deep Learning-Based Approach for Automatic VR Image Upright Adjustment

Jung, Raehyuk; Lee, Aiden Seuna Joon; Ashtari, Amirsaman; Bazin, Jean-Charles

doi:10.1109/vr.2019.8798326

Cited by 22 publications

(19 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the input of the CNN network is a set of perspective images sampled from the spherical image, the network model cannot learn the features from spherical images. The work of Jung et al [20] is the research that is the most similar to our work. In [20], the input of the CNN model is an equirectangular image, and the output is the azimuth and elevation angles relative to the upright image.…”

Section: Inclination Estimation Of Spherical Imagessupporting

confidence: 72%

“…The work of Jung et al [20] is the research that is the most similar to our work. In [20], the input of the CNN model is an equirectangular image, and the output is the azimuth and elevation angles relative to the upright image. In Section IV, a comparative experiment of our method and this method is presented.…”

Section: Inclination Estimation Of Spherical Imagessupporting

confidence: 72%

“…In this paper, we divide the estimation methods for image inclination into two categories, traditional feature-based methods [21], [34]- [38] and deep-learning-based methods [17], [19], [20], [33], [ On the other hand, deep-learning-based methods do not require such assumptions. For the deep-learning-based spherical image inclination estimation methods, there are two works that are similar to our work.…”

Section: Inclination Estimation Of Spherical Imagesmentioning

confidence: 99%

“…While a conventional camera that captures perspective images originates from the pin-hole camera model, a spherical camera that captures spherical images is represented by the spherical camera model. Spherical images are widely used and have been studied in the fields of medical science, such as representation of the retinal images of the eyes of humans [22], [49], geography, such as representation of the earth [43], meteorology, such as computation of atmospheric motion [42], and computer vision, such as immersive virtual reality [20], [26], [27], visual surveillance [29], augmented reality [25] and robotics [23], [24], [28].…”

Section: Introduction a Background 1) Pin-hole Camera Model Vs Smentioning

confidence: 99%

See 3 more Smart Citations

Discrete Spherical Image Representation for CNN-Based Inclination Estimation

Shan

2020

IEEE Access

View full text Add to dashboard Cite

How an image is represented as the input of a convolutional neural network (CNN) is important because this input directly influences the performance of the CNN. In this paper, we investigate the representation of spherical images by focusing on the inclination estimation of a spherical camera. Unlike other approaches to CNN-based inclination estimation, a spherical image is represented as a geodesicdivision-based discrete spherical image (DSI) that is obtained by sampling a sphere as uniformly as possible. The input of the CNN is a single image that consists of five parallelograms flattened from a regular icosahedron. To demonstrate the advantage of the proposed method, comparative experiments are conducted with two other spherical image representations, namely, equirectangular projection (ERP) and cubemap projection (CMP). The experimental results show that the proposed method using a geodesic-division-based discrete spherical image as the CNN input obtains the best performance-better than that of the cubemap and far superior to that of the equirectangular image. The effect of the image representations used becomes more significant as the relative inclination decreases. Moreover, comparative experiments are conducted using the state-of-the-art methods for spherical camera inclination compensation to further illustrate the superiority of the DSI representation. Consequently, the proposed method provides an important reference for the development of CNNs intended for spherical images.

show abstract

Section: Inclination Estimation Of Spherical Imagessupporting

confidence: 72%

Section: Inclination Estimation Of Spherical Imagessupporting

confidence: 72%

Section: Inclination Estimation Of Spherical Imagesmentioning

confidence: 99%

Section: Introduction a Background 1) Pin-hole Camera Model Vs Smentioning

confidence: 99%

See 2 more Smart Citations

Discrete Spherical Image Representation for CNN-Based Inclination Estimation

Shan

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Obtained panorama content with a 360 • fieldof-view sometimes needs to be further processed, for example due to unsuitable camera setup during data acquisition: e.g., if the cameras are tilted when capturing data, the stitched result will be misoriented. Jung et al [48] proposed a CNN-based method to predict the elevation and azimuth angles of the up-vector for a given VR image. To support model training, a large scale dataset of VR images with different orientations was generated, by combining random rotations and resizing of high resolution VR images from the SUN360 dataset [49].…”

Section: Panoramic Image and Video Creationmentioning

confidence: 99%

VR content creation and exploration with deep learning: A survey

Wang

Lyu

et al. 2020

Comp. Visual Media

View full text Add to dashboard Cite

Virtual reality (VR) offers an artificial, computer generated simulation of a real life environment. It originated in the 1960s and has evolved to provide increasing immersion, interactivity, imagination, and intelligence. Because deep learning systems are able to represent and compose information at various levels in a deep hierarchical fashion, they can build very powerful models which leverage large quantities of visual media data. Intelligence of VR methods and applications has been significantly boosted by the recent developments in deep learning techniques. VR content creation and exploration relates to image and video analysis, synthesis and editing, so deep learning methods such as fully convolutional networks and general adversarial networks are widely employed, designed specifically to handle panoramic images and video and virtual 3D scenes. This article surveys recent research that uses such deep learning methods for VR content creation and exploration. It considers the problems involved, and discusses possible future directions in this active and emerging research area. Keywords virtual reality; deep learning; neural networks; 360 • image and video virtual content

show abstract