Adaptive Foveated Rendering and Offloading in an Edge-Assisted Virtual Reality System

Abstract

Foveated rendering (FR)-in which the central foveal layer (the area around the eye gaze) of a virtual reality (VR) image is rendered at the highest resolution and the peripheral layers are rendered at progressively lower resolutions-is an advanced VR technique that controls the balance between computational load and visual quality by adjusting the foveal layer sizes. We consider an edge computingassisted VR computation offloading system incorporating FR and develop a deep reinforcement learning (DRL)-based solution to maximize a unified objective combining the visual quality and the overheads of energy consumption and delay for multiple VR users by optimizing the foveal layer size determination, offloading decisions and radio resource allocation of wireless links with non-orthogonal multiple access (NOMA). To formulate the unified objective, the user-perceived visual quality of a VR image rendered via FR is modeled with the sizes of the foveal layers as variables. The proposed solution consists of three main modules: per-user FR and offloading modules, which determine the sizes of the foveal layers and make offloading decisions, including subband allocation, and a central transmit-power allocation module. The actions of these three modules determine the reward that is in turn fed back to each module. Evaluation results reveal that the proposed framework can better adapt the operations of FR, offloading and wireless transmission to the environmental conditions than other FR and offloading benchmarks in terms of overall reward, visual quality, energy consumption and delay.INDEX TERMS virtual reality, foveated rendering, edge computing, offloading, deep reinforcement learning.