LSTM-Based Viewpoint Prediction for Multi-Quality Tiled Video Coding in Virtual Reality Streaming

Jamali, Mohammadreza; Coulombe, Stéphane; Vakili, Ahmad; Vázquez, Carlos

doi:10.1109/iscas45731.2020.9180528

Cited by 12 publications

(7 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Trajectory-based approaches [12]- [16], [22]- [24], [40]- [42] predict future viewing direction from one user's (single-user) or other users' (cross-user) historical head movement trajectories. [12], [40]- [42] proposed to use historical head movement data to predict future FoV.…”

Section: B Trajectory-based Prediction Methodsmentioning

confidence: 99%

Spherical Convolution empowered FoV Prediction in 360-degree Video Multicast with Limited FoV Feedback

Li¹,

Liu²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Field of view (FoV) prediction is critical in 360-degree video multicast, which is a key component of the emerging Virtual Reality (VR) and Augmented Reality (AR) applications. Most of the current prediction methods combining saliency detection and FoV information neither take into account that the distortion of projected 360degree videos can invalidate the weight sharing of traditional convolutional networks, nor do they adequately consider the difficulty of obtaining complete multi-user FoV information, which degrades the prediction performance. This paper proposes a spherical convolution-empowered FoV prediction method, which is a multi-source prediction framework combining salient features extracted from 360degree video with limited FoV feedback information. A spherical convolution neural network (CNN) is used instead of a traditional two-dimensional CNN to eliminate the problem of weight sharing failure caused by video projection distortion. Specifically, salient spatial-temporal features are extracted through a spherical convolution-based saliency detection model, after which the limited feedback FoV information is represented as a time-series model based on a spherical convolution-empowered gated recurrent unit network. Finally, the extracted salient video features are combined to predict future user FoVs. The experimental results show that the performance of the proposed method is better than other prediction methods.

show abstract

Section: B Trajectory-based Prediction Methodsmentioning

confidence: 99%

Spherical Convolution empowered FoV Prediction in 360-degree Video Multicast with Limited FoV Feedback

Li¹,

Liu²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…1) Content-Agnostic Approaches: Several existing contentagnostic approaches predict future viewing position using various prediction methods such as, average [17], linear regression (LR) [17], [178], Dead Reckoning (DR) [179], clustering [177], [180], [181], straightforward machine learning (ML) [182]- [184], and encoder-decoder architecture [183], [185]. Qian et al [17] used average, linear regression, and weighted linear regression models for viewport prediction and then entirely streamed those tiles that will overlap with the estimated viewport.…”

Section: A Viewport Predictionmentioning

confidence: 99%

A Survey on Adaptive 360° Video Streaming: Solutions, Challenges and Opportunities

Yaqoob

Muntean

2020

IEEE Commun. Surv. Tutorials

165

View full text Add to dashboard Cite

Omnidirectional or 360 • video is increasingly being used, mostly due to the latest advancements in immersive Virtual Reality (VR) technology. However, its wide adoption is hindered by the higher bandwidth and lower latency requirements than associated with traditional video content delivery. Diverse researchers propose and design solutions that help support an immersive visual experience of 360 • video, primarily when delivered over a dynamic network environment. This paper presents the state-of-the-art on adaptive 360 • video delivery solutions considering end-to-end video streaming in general and then specifically of 360 • video delivery. Current and emerging solutions for adaptive 360 • video streaming, including viewport-independent, viewport-dependent, and tile-based schemes are presented. Next, solutions for network-assisted unicast and multicast streaming of 360 • video content are discussed. Different research challenges for both on-demand and live 360 • video streaming are also analyzed. Several proposed standards and technologies and top international research projects are then presented. We demonstrate the ongoing standardization efforts for 360 • media services that ensure interoperability and immersive media deployment on a massive scale. Finally, the paper concludes with a discussion about future research opportunities enabled by 360 • video.

show abstract

“…Jiang et al [3] applied a model based on long short-term memory (LSTM) to predict future head rotations. Jamali et al [4] used a LSTM encoder-decoder network to perform sequence-to-sequence prediction. Nasrabadi et al [5] proposed a clustering-based method to estimate the user's future viewport.…”

Section: Viewport Predictionmentioning

confidence: 99%

“…e most advanced VR streaming research mainly focuses on viewport prediction [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16]. e existing solutions [1,[11][12][13] suggest prefetching all the tiles of each segment, and higher quality of prefetching predicts the tiles in the viewport.…”

Section: Introductionmentioning

confidence: 99%

“…e existing solutions [1,[11][12][13] suggest prefetching all the tiles of each segment, and higher quality of prefetching predicts the tiles in the viewport. Most existing viewport prediction algorithms can be divided into trajectory-based [1][2][3][4][5][6] and content-based [7][8][9][10][11][12][13][14][15][16] methods. However, these methods are either difficult to achieve sufficient prediction accuracy, or they occupy a large amount of server resources in actual deployment and cannot achieve excellent real-time performance due to the complexity of their own algorithms.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Lightweight Neural Network-Based Viewport Prediction for Live VR Streaming in Wireless Video Sensor Network

Chen

Cao

Ahmad

2021

Mobile Information Systems

View full text Add to dashboard Cite

Live virtual reality (VR) streaming (a.k.a., 360-degree video streaming) has become increasingly popular because of the rapid growth of head‐mounted displays and 5G networking deployment. However, the huge bandwidth and the energy required to deliver live VR frames in the wireless video sensor network (WVSN) become bottlenecks, making it impossible for the application to be deployed more widely. To solve the bandwidth and energy challenges, VR video viewport prediction has been proposed as a feasible solution. However, the existing works mainly focuses on the bandwidth usage and prediction accuracy and ignores the resource consumption of the server. In this study, we propose a lightweight neural network-based viewport prediction method for live VR streaming in WVSN to overcome these problems. In particular, we (1) use a compressed channel lightweight network (C-GhostNet) to reduce the parameters of the whole model and (2) use an improved gate recurrent unit module (GRU-ECA) and C-GhostNet to process the video data and head movement data separately to improve the prediction accuracy. To evaluate the performance of our method, we conducted extensive experiments using an open VR user dataset. The experiments results demonstrate that our method achieves significant server resource saving, real-time performance, and high prediction accuracy, while achieving low bandwidth usage and low energy consumption in WVSN, which meets the requirement of live VR streaming.

show abstract

LSTM-Based Viewpoint Prediction for Multi-Quality Tiled Video Coding in Virtual Reality Streaming

Cited by 12 publications

References 13 publications

Spherical Convolution empowered FoV Prediction in 360-degree Video Multicast with Limited FoV Feedback

Spherical Convolution empowered FoV Prediction in 360-degree Video Multicast with Limited FoV Feedback

A Survey on Adaptive 360° Video Streaming: Solutions, Challenges and Opportunities

Lightweight Neural Network-Based Viewport Prediction for Live VR Streaming in Wireless Video Sensor Network

Contact Info

Product

Resources

About