Extended TimeWarp latency compensation for virtual reality

Evangelakos, Daniel; Mara, Michael W.

doi:10.1145/2856400.2876015

Cited by 18 publications

(12 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Missing the rendering deadline will cause frame drop which significantly damages VR quality. Although the VR vendors today employ frame re-projection technologies such as Asynchronous Time Warp (ATW) [15,36] to artificially fill in dropped frames, they cannot fundamentally solve the problem of rendering deadline missing due to little consideration on users' perception and interaction. Thus, improving the overall rendering efficiency is still the highest design priority for modern VR-oriented GPUs [6].…”

Section: Smp Featured Gpu Architecturesmentioning

confidence: 99%

Oo-Vr

Xie

Chen

et al. 2019

Proceedings of the 46th International Symposium on Computer Architecture

View full text Add to dashboard Cite

With the strong computation capability, NUMA-based multi-GPU system is a promising candidate to provide sustainable and scalable performance for Virtual Reality (VR) applications and deliver the excellent user experience. However, the entire multi-GPU system is viewed as a single GPU under the single programming model which greatly ignores the data locality among VR rendering tasks during the workload distribution, leading to tremendous remote memory accesses among GPU models (GPMs). The limited inter-GPM link bandwidth (e.g., 64GB/s for NVlink) becomes the major obstacle when executing VR applications in the multi-GPU system. By conducting comprehensive characterizations on different kinds of parallel rendering frameworks, we observe that distributing the rendering object along with its required data per GPM can reduce the inter-GPM memory accesses. However, this object-level rendering still faces two major challenges in NUMA-based multi-GPU system: (1) the large data locality between the left and right views of the same object and the data sharing among different objects and (2) the unbalanced workloads induced by the softwarelevel distribution and composition mechanisms.To tackle these challenges, we propose object-oriented VR rendering framework (OO-VR) that conducts the software and hardware co-optimization to provide a NUMA friendly solution for VR multi-view rendering in NUMA-based multi-GPU systems. We first propose an object-oriented VR programming model to exploit the data sharing between two views of the same object and group objects into batches based on their texture sharing levels. Then, we design an object aware runtime batch distribution engine and distributed hardware composition unit to achieve the balanced workloads among GPMs and further improve the performance of VR rendering. Finally, evaluations on our VR featured simulator show that OO-VR provides 1.58x overall performance improvement

show abstract

Section: Smp Featured Gpu Architecturesmentioning

confidence: 99%

Oo-Vr

Xie

Chen

et al. 2019

Proceedings of the 46th International Symposium on Computer Architecture

View full text Add to dashboard Cite

show abstract

“…Late stage rendering. To reduce perceived latency, late stage rendering or time-warping [Evangelakos and Mara 2016;Van Waveren 2016] can be used. It is mostly studied in the context of AR/VR rendering where head-motion occurs after rendering and needs to be compensated for prior to display to reduce perceived latency.…”

Section: Related Workmentioning

confidence: 99%

Depth from motion for smartphone AR

et al. 2018

View full text Add to dashboard Cite

pipeline for mobile depth that supports a wide array of mobile phones, and uses only the existing monocular color sensor. Through several technical contributions, we provide the ability to compute low latency dense depth maps using only a single CPU core of a wide range of (medium-high) mobile phones. We demonstrate the capabilities of our approach on high-level AR applications including real-time navigation and shopping.

show abstract

“…Latency is an important factor when dealing with real-time experiences. Instead of running the neural re-rendering sequentially with the actual display update, we implemented a late stage reprojection phase [Evangelakos and Mara 2016;Van Waveren 2016]. In particular, we keep the computational stream of the network decoupled from the actual rendering, and use the current head pose to warp the final images accordingly.…”

Section: Real-time Free Viewpoint Neural Re-renderingmentioning

confidence: 99%

LookinGood

et al. 2018

View full text Add to dashboard Cite

Fig. 1. LookinGood leverages recent advances in real-time 3D performance capture and machine learning to re-render high quality novel viewpoints of a captured scene. A textured 3D reconstruction is first rendered to a novel viewpoint. Due to imperfections in geometry and low-resolution texture, the 2D rendered image contains artifacts and is low quality. Therefore we propose a deep learning technique that takes these images as input and generates more visually enhanced re-rendering. The system is specifically designed for VR and AR headsets, and accounts for consistency between two stereo views.Motivated by augmented and virtual reality applications such as telepresence, there has been a recent focus in real-time performance capture of humans under motion. However, given the real-time constraint, these systems often suffer from artifacts in geometry and texture such as holes and noise in the final rendering, poor lighting, and low-resolution textures. We take the novel approach to augment such real-time performance capture systems with a deep architecture that takes a rendering from an arbitrary viewpoint, and jointly performs completion, super resolution, and denoising of the imagery in real-time. We call this approach neural (re-)rendering, and * Authors equally contributed to this work. our live system "LookinGood". Our deep architecture is trained to produce high resolution and high quality images from a coarse rendering in real-time. First, we propose a self-supervised training method that does not require manual ground-truth annotation. We contribute a specialized reconstruction error that uses semantic information to focus on relevant parts of the subject, e.g. the face. We also introduce a salient reweighing scheme of the loss function that is able to discard outliers. We specifically design the system for virtual and augmented reality headsets where the consistency between the left and right eye plays a crucial role in the final user experience. Finally, we generate temporally stable results by explicitly minimizing the difference between two consecutive frames. We tested the proposed system in two different scenarios: one involving a single RGB-D sensor, and upper body reconstruction of an actor, the second consisting of full body 360 • capture. Through extensive experimentation, we demonstrate how our system generalizes across unseen sequences and subjects. The supplementary video is available at http://youtu.be/Md3tdAKoLGU.

show abstract

Extended TimeWarp latency compensation for virtual reality

Cited by 18 publications

References 1 publication

Oo-Vr

Oo-Vr

Depth from motion for smartphone AR

LookinGood

Contact Info

Product

Resources

About