Predicting head trajectories in 360° virtual reality videos

Aladagli, A. Deniz; Ekmekcioǧlu, Erhan; Jarnikov, Dmitri; Kondoz, A.M.

doi:10.1109/ic3d.2017.8251913

Cited by 29 publications

(18 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Azuma et al [16] characterized the user's head motion as position, velocity and acceleration, and proposed a predictor to derive the future head position. The authors in [17] took content-related features into account, and predicted the viewpoint based on a saliency algorithm.…”

Section: Related Workmentioning

confidence: 99%

Probabilistic Tile Visibility-Based Server-Side Rate Adaptation for Adaptive 360-Degree Video Streaming

Liu

Yang

et al. 2020

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

In this paper, we study the server-side rate adaptation problem for streaming tile-based adaptive 360-degree videos to multiple users who are competing for transmission resources at the network bottleneck. Specifically, we develop a convolutional neural network (CNN)-based viewpoint prediction model to capture the nonlinear relationship between the future and historical viewpoints. A Laplace distribution model is utilized to characterize the probability distribution of the prediction error. Given the predicted viewpoint, we then map the viewport in the spherical space into its corresponding planar projection in the 2-D plane, and further derive the visibility probability of each tile based on the planar projection and the prediction error probability. According to the visibility probability, tiles are classified as viewport, marginal and invisible tiles. The server-side tile rate allocation problem for multiple users is then formulated as a non-linear discrete optimization problem to minimize the overall received video distortion of all users and the quality difference between the viewport and marginal tiles of each user, subject to the transmission capacity constraints and users' specific viewport requirements.We develop a steepest descent algorithm to solve this non-linear discrete optimization problem, by initializing the feasible starting point in accordance with the optimal solution of its continuous relaxation.Extensive experimental results show that the proposed algorithm can achieve a near-optimal solution, and outperforms the existing rate adaptation schemes for tile-based adaptive 360-video streaming. Index Terms360-degree video, tile-based adaptive streaming, server-side rate adaptation, viewpoint/viewport prediction, tile visibility probability

show abstract

Section: Related Workmentioning

confidence: 99%

Probabilistic Tile Visibility-Based Server-Side Rate Adaptation for Adaptive 360-Degree Video Streaming

Liu

Yang

et al. 2020

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

show abstract

“…However, these approaches in [4], [5] use naive models and ignore video content's relation to future movement, thus are less accurate. Other existing works such as in [6]- [10] combine both the video content features and orientation of HMD to predict the future head movement. In [6], the authors use a pre-trained saliency model to predict head movement.…”

Section: A Related Workmentioning

confidence: 99%

“…Although the works in [6]- [10] use both video saliency and history head orientation to predict future head movement, these prior works do not directly treat video frame content in much detail. Since VR videos contain various scenes, and further, each scene has different regions of interest for users.…”

Section: A Related Workmentioning

confidence: 99%

Deep Learning for Content-Based Personalized Viewport Prediction of 360-Degree VR Videos

2020

View full text Add to dashboard Cite

In this paper, the problem of head movement prediction for virtual reality videos is studied. In the considered model, a deep learning network is introduced to leverage position data as well as video frame content to predict future head movement. For optimizing data input into this neural network, data sample rate, reduced data, and long-period prediction length are also explored for this model. Simulation results show that the proposed approach yields 16.1% improvement in terms of prediction accuracy compared to a baseline approach that relies only on the position data.

show abstract

“…The strategy presented by Aladagli et al in [1] extracts the saliency from the current frame with an off-the-shelf method, identifies the most salient point, and predicts the next FoV to be centered on this most salient point. It then builds recursively.…”

Section: Ic3d17mentioning

confidence: 99%

A unified evaluation framework for head motion prediction methods in 360° videos

Rondón

Sassatelli

Aparicio-Pardo

et al. 2020

Proceedings of the 11th ACM Multimedia Systems Conference

View full text Add to dashboard Cite

The streaming transmissions of 360°videos is a major challenge for the development of Virtual Reality, and require a reliable head motion predictor to identify which region of the sphere to send in high quality and save data rate. Different head motion predictors have been proposed recently. Some of these works have similar evaluation metrics or even share the same dataset, however, none of them compare with each other. In this article we introduce an open software that enables to evaluate heterogeneous head motion prediction methods on various common grounds. The goal is to ease the development of new head/eye motion prediction methods. We first propose an algorithm to create a uniform data structure from each of the datasets. We also provide the description of the algorithms used to compute the saliency maps either estimated from the raw video content or from the users' statistics. We exemplify how to run existing approaches on customizable settings, and finally present the targeted usage of our open framework: how to train and evaluate a new prediction method, and compare it with existing approaches and baselines in common settings. The entire material (code, datasets, neural network weights and documentation) is publicly available. CCS CONCEPTS • Computing methodologies → Model verification and validation; Virtual reality.

show abstract

Predicting head trajectories in 360° virtual reality videos

Cited by 29 publications

References 16 publications

Probabilistic Tile Visibility-Based Server-Side Rate Adaptation for Adaptive 360-Degree Video Streaming

Probabilistic Tile Visibility-Based Server-Side Rate Adaptation for Adaptive 360-Degree Video Streaming

Deep Learning for Content-Based Personalized Viewport Prediction of 360-Degree VR Videos

A unified evaluation framework for head motion prediction methods in 360° videos

Contact Info

Product

Resources

About