User-Adaptive Editing for 360 degree Video Streaming with Deep Reinforcement Learning

Abstract

The development through streaming of 360°videos is persistently hindered by how much bandwidth they require. Adapting spatially the quality of the sphere to the user's Field of View (FoV) lowers the data rate but requires to keep the playback buffer small, to predict the user's motion or to make replacements to keep the buffered qualities up to date with the moving FoV, all three being uncertain and risky. We have previously shown that opportunistically regaining control on the FoV with active attention-driving techniques makes for additional levers to ease streaming and improve Quality of Experience (QoE). Deep neural networks have been recently shown to achieve best performance for video streaming adaptation and head motion prediction. This demo presents a step ahead in the important investigation of deep neural network approaches to obtain user-adaptive and network-adaptive 360°video streaming systems. In this demo, we show how snap-changes, an attentiondriving technique, can be automatically modulated by the user's motion to improve the streaming QoE. The control of snap-changes is made with a deep neural network trained on head motion traces with the Deep Reinforcement Learning strategy A3C. CCS CONCEPTS • Human-centered computing → Virtual reality; • Networks → Network simulations; • Computing methodologies → Neural networks.

Towards the Effects of Alignment Edits on the Quality of Experience of 360 Videos

“…For the aligned group we observe that FR 40 has the lower mean: 8% lower than snapcut, followed by FR 60 • with 2.46% lower than snap-cut, the total mean head speed reduction was FR 10 • = 14.9 • , FR 20 • = 9.5, FR 40 • = 26.7 • , FR 60 • =33.1 • , Snap-cut = 21.5 • . A common consequence between all edit types tested is that they affected reducing head speed, which may be related to the fixation at ROI, reducing the exploratory behavior, this stability effect agrees with the literature [79] and can be useful for streaming applications. As a remark, for this analysis we filtering head speed higher than 150 • /s (refer to Section IV-C3 for description).…”