Streaming 360-Degree Videos Using Super-Resolution

Dasari, Mallesham; Bhattacharya, Arani; Vargas, Santiago; Sahu, Pranjal; Balasubramanian, Aruna; Das, Samir R.

doi:10.1109/infocom41043.2020.9155477

Cited by 107 publications

(38 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Content-based methods use rotation information and video content as features. Dasari et al [7] proposed a PARSEC method that combines viewport prediction with superresolution to improve the quality of user experience. Kan et al [8] proposed a DRL-based rate adaptation algorithm qualified for learning a policy to strike an optimal trade-off between the video quality, the risk of rebuffering, and the smoothness of video quality by selecting the proper bitrate of tiles.…”

Section: Viewport Predictionmentioning

confidence: 99%

“…e most advanced VR streaming research mainly focuses on viewport prediction [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16]. e existing solutions [1,[11][12][13] suggest prefetching all the tiles of each segment, and higher quality of prefetching predicts the tiles in the viewport.…”

Section: Introductionmentioning

confidence: 99%

“…e existing solutions [1,[11][12][13] suggest prefetching all the tiles of each segment, and higher quality of prefetching predicts the tiles in the viewport. Most existing viewport prediction algorithms can be divided into trajectory-based [1][2][3][4][5][6] and content-based [7][8][9][10][11][12][13][14][15][16] methods. However, these methods are either difficult to achieve sufficient prediction accuracy, or they occupy a large amount of server resources in actual deployment and cannot achieve excellent real-time performance due to the complexity of their own algorithms.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Lightweight Neural Network-Based Viewport Prediction for Live VR Streaming in Wireless Video Sensor Network

Chen

Cao

Ahmad

2021

Mobile Information Systems

View full text Add to dashboard Cite

Live virtual reality (VR) streaming (a.k.a., 360-degree video streaming) has become increasingly popular because of the rapid growth of head‐mounted displays and 5G networking deployment. However, the huge bandwidth and the energy required to deliver live VR frames in the wireless video sensor network (WVSN) become bottlenecks, making it impossible for the application to be deployed more widely. To solve the bandwidth and energy challenges, VR video viewport prediction has been proposed as a feasible solution. However, the existing works mainly focuses on the bandwidth usage and prediction accuracy and ignores the resource consumption of the server. In this study, we propose a lightweight neural network-based viewport prediction method for live VR streaming in WVSN to overcome these problems. In particular, we (1) use a compressed channel lightweight network (C-GhostNet) to reduce the parameters of the whole model and (2) use an improved gate recurrent unit module (GRU-ECA) and C-GhostNet to process the video data and head movement data separately to improve the prediction accuracy. To evaluate the performance of our method, we conducted extensive experiments using an open VR user dataset. The experiments results demonstrate that our method achieves significant server resource saving, real-time performance, and high prediction accuracy, while achieving low bandwidth usage and low energy consumption in WVSN, which meets the requirement of live VR streaming.

show abstract

Section: Viewport Predictionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Lightweight Neural Network-Based Viewport Prediction for Live VR Streaming in Wireless Video Sensor Network

Chen

Cao

Ahmad

2021

Mobile Information Systems

View full text Add to dashboard Cite

show abstract

“…While most of the existing studies focus on SR technique itself, not many have yet applied the SR to the practical video streaming and/or content delivery networks and jointly optimized the network decisions (e.g., bitrate adaptation, scheduling, and data transmission) together. Recently, on-device SR computation methods, which enable the enhancement of the video quality independent at the receiver have been proposed in [35]- [37]. Specifically, Dejavu in [35] enhances the videoconferencing in real-time by employing the historical sessions.…”

Section: Deep Learning-based Super-resolutionmentioning

confidence: 99%

Delay-Sensitive and Power-Efficient Quality Control of Dynamic Video Streaming using Adaptive Super-Resolution

Choi¹,

Yun²,

Kim³

2021

Preprint

View full text Add to dashboard Cite

In a decade, the adaptive quality control of video streaming and the super-resolution (SR) technique have been deeply explored. As edge devices improved to have exceptional processing capability than ever before, streaming users can enhance the received image quality to allow the transmitter to compress the images to save its power or pursue network efficiency. In this sense, this paper proposes a novel dynamic video streaming algorithm that adaptively compresses video chunks at the transmitter and separately enhances the quality at the receiver using SR. In order to allow transmission of video chunks with different compression levels and control of the computation burden, we present the adaptive SR network which is optimized by minimizing the weighted sum of losses extracted from different layer outputs. for dynamic video streaming. In addition, we jointly orchestrate video delivery and resource usage, and the proposed video delivery scheme balances the tradeoff well among the average video quality, the queuing delay, buffering time, transmit power, and computation power. Simulation results show that the proposed scheme pursues the quality-of-services (QoS) of the video streaming better than the adaptive quality control without the cooperation of the transmitter and the receiver and the non-adaptive SR network.

show abstract

“…SR was also used in previous work related to VR using 360-video [24], here, the size of the models were addressed by using micro-models that cover short video segments.…”

Section: B Super Resolution (Sr)mentioning

confidence: 99%

Remapping of Hidden Area Mesh Pixels for Codec Speed-up in Remote VR

Kelkkanen

Fiedler

Lindero

2021

2021 13th International Conference on Quality of Multimedia Experience (QoMEX)

View full text Add to dashboard Cite

Rendering VR-content generally requires large image resolutions. This is both due to the display being positioned close to the eyes of the user and to the super-sampling typically used in VR. Due to the requirements of low latency and large resolutions in VR, remote rendering can be diffcult t o support at suffcient s peeds i n t his medium.In this paper, we propose a method that can reduce the required resolution of non-panoramic VR images from a codec perspective. Because VR images are viewed close-up from within a headset with specifc l enses, t here a re r egions o f t he images that will remain unseen by the user. This unseen area is referred to as the Hidden-Area Mesh (HAM) and makes up 19% of the screen on the HTC Vive VR headset as one example. By remapping the image in a specifc m anner, w e c an c ut o ut the HAM, reduce the resolution by the size of the mesh and thus reduce the amount of data that needs to be processed by encoder and decoder. Results from a prototype remote renderer show that by using the proposed Hidden-Area Mesh Remapping (HAMR), an implementation-dependent speed-up of 10-13% in encoding, 17-18% in decoding and 7-11% in total can be achieved while the negative impact on objective image quality in terms of SSIM and VMAF remains small.

show abstract

Streaming 360-Degree Videos Using Super-Resolution

Cited by 107 publications

References 30 publications

Lightweight Neural Network-Based Viewport Prediction for Live VR Streaming in Wireless Video Sensor Network

Lightweight Neural Network-Based Viewport Prediction for Live VR Streaming in Wireless Video Sensor Network

Delay-Sensitive and Power-Efficient Quality Control of Dynamic Video Streaming using Adaptive Super-Resolution

Remapping of Hidden Area Mesh Pixels for Codec Speed-up in Remote VR

Contact Info

Product

Resources

About