Complexity-Reduced Super Resolution for Foveation-Based Driving Head Mounted Displays

Abstract

In this paper, we propose a foveation-based super resolution (SR) algorithm to create high resolution images from low resolution inputs for virtual reality head mounted displays. Because the proposed SR algorithm is integrated in the previous foveation-based driving technology to cover the small area around the foveation point that requires high rendering quality, the overall computational complexity is substantially reduced, compared to the whole area SR. The target display has 4 times as high resolution as the input image, therefore, the proposed SR algorithm generates 4× as well as 2× SR images at the same time. To support two SR output images, small area SR, and small number of weights, we employ cropping as well as progressive and recursive framework used in the previous MS-LapSRN. We reduce the number of neurons by placing the deconvolutional layer after convolutional layers, compared to the MS-LapSRN. PSNR and SSIM performances of the proposed SR for the 4× scale are estimated as 31.152 dB and 0.935 for Set5, 26.656 dB and 0.858 for Set14, 27.138 dB and 0.830 for BSD100, and 25.078 dB and 0.836 for Urban100. For the target 8K display of 7,680×4,320, the proposed FovSR-integrated driving technology achieves the substantial reductions by 76.7 % and 99.02 % on the number of lines from 7,680 to 1,788 and the number of neurons from 24,518,246,400 to 239,541,184, respectively.

Review of Integrated Gate Driver Circuits in Active Matrix Thin-Film Transistor Display Panels

“…To address this issue, the foveation-based driving scheme has been proposed to reduce the effective number of lines in a frame by reducing the vertical image resolution of distant regions from the fixation point [111,112]. It is the high resolution display driving technique to realize the foveated-rendering images directly in a panel as presented in Figure 18.…”

Neural Foveated Super-Resolution for Real-time VR Rendering

“…It has to train different networks separately to deal with different foveation layers. FOCAS 53 and FovMSLapSRN (FovMLS) 54 also target on foveated super-resolution. They generate image regions with different qualities, either with the partial model or recursive neural network, respectively.…”

“…Additionally, we included the EDSR 21 baseline, which is widely used and performs the second best according to the evaluation conducted by Xiao et al 5 . To explore the effectiveness of the classical neural networks that have been applied in the context of foveated rendering 53,54 , we compared our method with MSLapSRN (MLS) 58 and RRN 27 . MLS utilizes a shared network module for recursive processing and step-by-step upscaling of low-resolution images while RRN is a classical recurrent network for video super-resolution.…”

“…In this section, we assess the quality of foveated super-resolution produced by our method and compare it with two superresolution methods focusing on generating foveated results: (1) FovMLS 54 , a method that leverages the characteristic of MLS 58 for foveated rendering; (2) FOCAS 53 , a method employs the recurrent networks of RRN 27 for foveated super-resolution. To evaluate the performance, we utilize test datasets consisting of sequences recorded in VR, capturing head motion and gaze information as described in Section 4.1.…”