Foveated streaming of real-time graphics

Abstract

Remote rendering systems comprise powerful servers that render graphics on behalf of low-end client devices and stream the graphics as compressed video, enabling high end gaming and Virtual Reality on those devices. One key challenge with them is the amount of bandwidth required for streaming high quality video. Humans have spatially non-uniform visual acuity: We have sharp central vision but our ability to discern details rapidly decreases with angular distance from the point of gaze. This phenomenon called foveation can be taken advantage of to reduce the need for bandwidth. In this paper, we study three different methods to produce a foveated video stream of real-time rendered graphics in a remote rendered system: 1) foveated shading as part of the rendering pipeline, 2) foveation as post processing step after rendering and before video encoding, 3) foveated video encoding. We report results from a number of experiments with these methods. They suggest that foveated rendering alone does not help save bandwidth. Instead, the two other methods decrease the resulting video bitrate significantly but they also have different quality per bit and latency profiles, which makes them desirable solutions in slightly different situations. CCS CONCEPTS• Computer systems organization → Real-time system architecture; • Computing methodologies → Image compression; Non-photorealistic rendering; Virtual reality; • Networks → Cloud computing.

An Introduction to Online Video Game QoS and QoE Influencing Factors

“…Particular interest resides also on the reduction of the high throughput dependence, primarily by tasking the client with more than just simple video decoding, e.g., moving parts of the game logic or rendering process back to the client. Examples of these approaches can be found in [235]- [238]. Although, none of these approaches has been realized in a practical system yet, as they can impose drastic overhead or interfere with game design and development decisions.…”