Fixation Prediction for 360° Video Streaming in Head-Mounted Virtual Reality

Fan, Ching-Ling; Lee, Jean; Lo, Wen-Chih; Huang, Chun-Ying; Chen, Kuan-Ta; Hsu, Cheng-Hsin

doi:10.1145/3083165.3083180

Cited by 192 publications

(119 citation statements)

References 15 publications

Supporting

Mentioning

119

Contrasting

Order By: Relevance

“…The additional use of techniques for view specification based on e.g. fixation prediction [10] result in additional delays of around 40ms which represents a noticeable perceivable lag in remote collaboration scenarios and reduces the interactive experience. In addition, when the reconstruction is finished or paused and the 3D model does not change for a certain time, the video stream of the renderings still requires a constantly high amount of bandwidth whereas the bandwidth required for streaming the 3D model would immediately drop to zero.…”

Section: Design Choicesmentioning

confidence: 99%

SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence

Stotko

Krumpen

Hullin

et al. 2019

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Fig. 1. Illustration of our novel multi-client live telepresence framework for remote collaboration: RGB-D data captured with consumergrade cameras represent the input to our real-time large-scale reconstruction technique that is based on a novel thread-safe GPU hash map data structure. Efficient data streaming is achieved by transmitting a novel compact representation of the reconstructed model in terms of Marching Cubes indices. Multi-client live telepresence is achieved by the server's independent handling of client requests.Abstract-Real-time 3D scene reconstruction from RGB-D sensor data, as well as the exploration of such data in VR/AR settings, has seen tremendous progress in recent years. The combination of both these components into telepresence systems, however, comes with significant technical challenges. All approaches proposed so far are extremely demanding on input and output devices, compute resources and transmission bandwidth, and they do not reach the level of immediacy required for applications such as remote collaboration. Here, we introduce what we believe is the first practical client-server system for real-time capture and many-user exploration of static 3D scenes. Our system is based on the observation that interactive frame rates are sufficient for capturing and reconstruction, and real-time performance is only required on the client site to achieve lag-free view updates when rendering the 3D model. Starting from this insight, we extend previous voxel block hashing frameworks by introducing a novel thread-safe GPU hash map data structure that is robust under massively concurrent retrieval, insertion and removal of entries on a thread level. We further propose a novel transmission scheme for volume data that is specifically targeted to Marching Cubes geometry reconstruction and enables a 90% reduction in bandwidth between server and exploration clients. The resulting system poses very moderate requirements on network bandwidth, latency and client-side computation, which enables it to rely entirely on consumer-grade hardware, including mobile devices. We demonstrate that our technique achieves state-of-the-art representation accuracy while providing, for any number of clients, an immersive and fluid lag-free viewing experience even during network outages.

show abstract

Section: Design Choicesmentioning

confidence: 99%

SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence

Stotko

Krumpen

Hullin

et al. 2019

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

show abstract

“…Saliency and motion map of videos are also available in the dataset. e authors designed a viewport prediction method based on their dataset in [6]. Wu et al [17] provided a dataset with 18 videos watched by 48 participants.…”

Section: Related Workmentioning

confidence: 99%

“…Only the videos that we have created 6. for all videos in the taxonomy, the overlaid version can be found in: h ps://bit.ly/2P1DRR7.…”

mentioning

confidence: 99%

A taxonomy and dataset for 360° videos

Nasrabadi

Samiei

Mahzari

et al. 2019

Proceedings of the 10th ACM Multimedia Systems Conference

View full text Add to dashboard Cite

In this paper, we propose a taxonomy for 360 • videos that categorizes videos based on moving objects and camera motion. We gathered and produced 28 videos based on the taxonomy, and recorded viewport traces from 60 participants watching the videos. In addition to the viewport traces, we provide the viewers' feedback on their experience watching the videos, and we also analyze viewport pa erns on each category.

show abstract

“…In addition to the aforementioned work [6]- [8] which mainly focus on the VR video-level design, researches on investigating the opportunities for mobile VR delivery that can be potentially obtained via efficiently utilizing resources at the mobile edge network (MEN), i.e., communications, computing and caching (3C), are also drawing increasing attention [4], [5], [9]- [11]. Specifically, [4] illustrates the potential gain obtained from utilizing the resources at the MEN via simulation results.…”

Section: Introduction and Frameworkmentioning

confidence: 97%

“…Then, by knowing user's FOV, researches on multi-view and tile-based video streaming are being investigated [7]. Moreover, in order to further improve the quality of experience for tile-based video streaming, motion-prediction-based transmission is also being studied based on dataset collected from real users [8].…”

Section: Introduction and Frameworkmentioning

confidence: 99%

Communication, Computing and Caching for Mobile VR Delivery: Modeling and Trade-Off

Sun

Chen

Tao

et al. 2018

2018 IEEE International Conference on Communications (ICC)

View full text Add to dashboard Cite

Mobile virtual reality (VR) delivery is gaining increasing attention from both industry and academia due to its ability to provide an immersive experience. However, achieving mobile VR delivery requires ultra-high transmission rate, deemed as a first killer application for 5G wireless networks. In this paper, in order to alleviate the traffic burden over wireless networks, we develop an implementation framework for mobile VR delivery by utilizing caching and computing capabilities of mobile VR device. We then jointly optimize the caching and computation offloading policy for minimizing the required average transmission rate under the latency and local average energy consumption constraints. In a symmetric scenario, we obtain the optimal joint policy and the closed-form expression of the minimum average transmission rate. Accordingly, we analyze the tradeoff among communication, computing and caching, and then reveal analytically the fact that the communication overhead can be traded by the computing and caching capabilities of mobile VR device, and also what conditions must be met for it to happen. Finally, we discuss the optimization problem in a heterogeneous scenario, and propose an efficient suboptimal algorithm with low computation complexity, which is shown to achieve good performance in the numerical results.

show abstract

Fixation Prediction for 360° Video Streaming in Head-Mounted Virtual Reality

Cited by 192 publications

References 15 publications

SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence

SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence

A taxonomy and dataset for 360° videos

Communication, Computing and Caching for Mobile VR Delivery: Modeling and Trade-Off

Contact Info

Product

Resources

About