Gaze-Aware Streaming Solutions for the Next Generation of Mobile VR Experiences

Abstract

This paper presents a novel approach to content delivery for video streaming services. It exploits information from connected eye-trackers embedded in the next generation of VR Head Mounted Displays (HMDs). The proposed solution aims to deliver high visual quality, in real time, around the users' fixations points while lowering the quality everywhere else. The goal of the proposed approach is to substantially reduce the overall bandwidth requirements for supporting VR video experiences while delivering high levels of user perceived quality. The prerequisites to achieve these results are: (1) mechanisms that can cope with different degrees of latency in the system and (2) solutions that support fast adaptation of video quality in different parts of a frame, without requiring a large increase in bitrate. A novel codec configuration, capable of supporting near-instantaneous video quality adaptation in specific portions of a video frame, is presented. The proposed method exploits in-built properties of HEVC encoders and while it introduces a moderate amount of error, these errors are indetectable by users. Fast adaptation is the key to enable gaze-aware streaming and its reduction in bandwidth. A testbed implementing gaze-aware streaming, together with a prototype HMD with in-built eye tracker, is presented and was used for testing with real users. The studies quantified the bandwidth savings achievable by the proposed approach and characterize the relationships between Quality of Experience (QoE) and network latency. The results showed that up to 83% less bandwidth is required to deliver high QoE levels to the users, as compared to conventional solutions.

Video Caching, Analytics, and Delivery at the Wireless Edge: A Survey and Future Directions

“…Such an approach allows the streaming service to maintain low latency of streaming and prevent VR sickness, while maximizing the quality of streaming. Lungaro et al [247] propose a gazeaware video streaming solution for 360°videos using a headmounted display with an eye tracker. The proposed solution utilizes a server for video tile provisioning and streaming that can be deployed in the edge-C3.…”

Taming the Latency in Multi-User VR 360°: A QoE-Aware Deep Learning-Aided Multicast Framework

“…Foveated, tile-based and projection/viewportbased FoV-streaming are the most commonly adopted approaches. Foveated solutions as per [5], [6] require advance eye-gaze tracking in the HMDs and real-time frame rendering in the servers. Whereas the main drawback of projectionstreaming [7] lies in its large server storage needs given that for each frame multiple viewpoints are kept.…”

“…Moreover, as stated before, users in U will report their 6DoF pose information in the UL with T f periodicity. With these assumptions, even if both the SBSs and users are aware of each other's location and know a priori what their respective beams ideal boresight directions are, the above antenna model effectively captures subtle misalignment errors arriving from the limited 6 availability of unique beampatterns common in codebookbased beam alignment approaches under analog beamforming. In other words, the angular deviation ϑ bu (t) in (7) reflects the inability of the analog beamformer to direct the mainbeam at any arbitrary location.…”

“…C (t) be the new expressions for the utilities which exploit the estimated service ratê 6 We remark here that by matching each SBS to a single cluster with orthogonal non-overlapping beams for the multicast transmission, only the impairment due to inter-SBS interference needs to be considered.…”

2018 Index IEEE Transactions on Visualization and Computer Graphics Vol. 24

“…Sjoberg, R., see1535-1544 Smilkov, D., see Wongsuphasawat, K.,TVCG Jan. 2018 1-12 Sondag, M., Speckmann, B., and Verbeek, K., Stable Treemaps via Local Moves; TVCG Jan. 2018 729-738 Sorger, J., see 1014-1024 Sorger, J., seeMindek, P., TVCG Jan. 2018 883-892 Speckmann, B., see Sondag, M.,TVCG Jan. 2018 729-738 Sperrle, F., see El-Assady, M.,TVCG Jan. 2018 382-391 Springer, D., see Huang, H.,TVCG Sept. 2018 2516-2530 Steiner, D., see Ballester-Ripoll, R., TVCG Oct. 2018 2714-2727 Steinicke, F., see Kiyokawa, K., TVCG April 2018 vi-vii Steinicke, F., see Zhang, J., TVCG April 2018 1671-1680 Steinicke, F., see 2663-2674 Stewart, W.F., seeKwon, B.C., TVCG Jan. 2018 142-151 Stitz, H., see Niederer, C.,TVCG Jan. 2018 677-686 Streit, M., see Niederer, C.,TVCG Jan. 2018 677-686 Strobelt, H., Gehrmann, S., Pfister, H., and Rush, A.M., LSTMVis: A Tool forVisual Analysis of Hidden State Dynamics in Recurrent Neural Networks; TVCG Jan. 2018 667-676 Su, Z., see2993 Subr, K., see Lyu, G., TVCG April 2018 1554-1563 Subramanian, K., seeBurlinson, D., TVCG Jan. 2018 574-583 Suga, C., seeMerenda, C., TVCG Nov. 2018 2875-2885 Sugimoto, M., seeHamasaki, T., TVCG April 2018 1457-1466, see Han, D.T., TVCG April 2018 1467-1476 Summers, A., see Bock, A.,TVCG Jan. 2018 812-821 Sun, G., see 2758-2772 Sun, H., see1260-1273, see Zhao, J., Sun, Y., see Wang, Y.,TVCG Jan. 2018 489-499 Sun, Y., Schaefer, S., and Wang, W., Image Structure Retrieval via L 0 Minimization; TVCG July 2018 2129-2139 Swan, J.E., see Broll, W., TVCG July 2018 2266-2267 Swan, J.E., see Grubert, J.,TVCG Sept. 2018 2649-2662 Swan II, J.E., see Broll, W., TVCG July 2018 2265 Szafir, D.A., Modeling Color Difference for Visualization Design; TVCG Jan. TVCGDec. 2018 3200-3213 Tao, J., and Wang, C., Semi-Automatic Generation of Stream Surfaces via Sketching; TVCG Sept. 2018 2622-2635 Tauzia, E., see 1368-1380 Telea, A., see Hurter, C., TVCG Jan. 2018 500-510 Telea, A.C., see Matute, J.,TVCG Jan. 2018 542-552 Teng, L., see 1554-1563 Teng, X., see Cao, N., TVCG Jan. 2018 Thalmann, D., see 2689-2701 Theisel, H., see 2424-2439 Thiagarajan, J.J., see 553-562 Thomas, B., see Kiyokawa, K., TVCG April 2018 vi-vii Thomas, B.H., see Piumsomboon, T., TVCG Nov. 2018 2974-2982 Thomas, B.H., see Volmer, B., TVCG Nov. 2018 2846-2856 Thomas, B.H., see Reichherzer, C., TVCG Nov. 2018 2917-2926 Throm, N., see Xiao, R., TVCG April 2018 1653-1660 Tian, S., see Wang, W., TVCG Oct. 2018 2787-2798 Tian, Y., see 2799-2812 832-842 Tollmar, K., see Lungaro, P., TVCG April 20181535-1544 Toloudis, D., seeMindek, P., TVCG Jan. 2018 883-892 Tong, Q., Yuan, Z., Liao, X., Zheng, M., Yuan, T., and Zhao, J., Magnetic LevitationHaptic Augmentation for Virtual Tissue Stiffness Perception; TVCG Dec. 2018 3123-3136 Tong, X., see Wang, Y., TVCG May 2018 1856-1866 Tong, Y., see Wang, X., TVCG Dec. 2018 3214-3224 Torr, P.H.S., see Golodetz, S., TVCG Nov. 2018 2895-2905 Tory, M., see Hoque, E., TVCG Jan. 2018 309-318 Tory, M., see Lam, H., TVCG Jan. 2018 435-445 Tost, B., see Kumpf, A., TVCG Jan. 2018 109-119 Tu, C., see Wang, Y., TVCG Dec. 2018 3096-3110 Tung, A.K., see Xia, J., TVCG Jan. 2018 236-245 Turton, T.L., see Bujack, R., TVCG Jan. 2018 923-933 Jayaraman, P.K., þ, TVCG July 2018 2103-2117 Assembling A Comparison of Virtual and Physical Training Transfer of Bimanual Assembly Tasks. Murcia-Lopez, M., þ, TVCG April 2018 1574-1583 Recovering Functional Mechanical Assemblies from Raw Scans.…”