Dynamic Adaptive Streaming for Augmented Reality Applications

Abstract

Augmented Reality (AR) superimposes digital content on top of the real world, to enhance it and provide a new generation of media experiences. To provide a realistic AR experience, objects in the scene should be delivered with both high photorealism and low latency. Current AR experiences are mostly delivered with a download-and-play strategy, where the whole scene is considered a monolithic entity for delivery. This approach results in high start-up latencies and therefore a poor user experience. A similar problem in the video domain has already been tackled with the HTTP Adaptive Streaming (HAS) principle, where the video is split into segments, and a rate adaptation heuristic dynamically adapts the video quality based on the available network resources. In this paper, we apply the adaptive streaming principle from the video to the AR domain, and propose a streaming framework for AR applications. In our proposed framework, the AR objects are available at different Levels-Of-Detail (LODs) and can be streamed independently from each other. An LOD adaptation heuristic is in charge of dynamically deciding what object should be fetched from the server and at what LOD level. Our proposed heuristic prioritizes content that is more likely to be viewed by the user and selects the best LOD to maximize the object's perceived visual quality. Moreover, the adaptation takes into account the available bandwidth resources to ensure a timely delivery of the AR objects. Experiments carried out over the Internet using an AR streaming prototype developed on an iOS device allow us to show the gains brought by the proposed framework. Particularly, our approach can decrease start-up latency up to 90% with respect to a download-and-play baseline, and decrease the amount of data needed to deliver the AR experience up to 79%, without sacrificing on the visual quality of the AR objects.

Progressive Mesh-based HTTP Adaptive Augmented Reality Streaming System

“…Next, the performance of the proposed system is compared with those of existing systems. namely, Full-Request, Round-Robin, DASH-NVE [11], and DAS-AR [10]. As the existing systems do not specify the mechanism interacting with the real world, the position of AR objects is determined by the proposed marker size and position detection processes for the existing systems.…”

“…So far, a considerable amount of research efforts has been devoted to provide adaptive AR streaming services with limited network resources. Petrangeli et al [10] proposed an AR streaming framework that dynamically decides the LOD chunk of 3D objects considering the geometric error and the position of the 3D objects. Forgione et al [11] introduced a DASH-based networked virtual environment while considering the user view and network conditions, and the number of triangles is considered to determine the download chunk of 3D con-tents.…”

Instant Reality: Gaze-Contingent Perceptual Optimization for 3D Virtual Reality Streaming

“…To incorporate the unique structures of 3D meshes, Hladky et al [15] prioritize the triangle components in the streaming pipeline based on their individual visibility. In a real cloud-based system, object-wise LoDs shall be determined with respect to the virtual camera views and the available resources (bandwidth) and constraints (reaction time) [33]. It is also possible to perform progressive rendering without building hierarchical structures [40].…”

Evaluating the Impact of Tiled User-Adaptive Real-Time Point Cloud Streaming on VR Remote Communication

“…In general, tiled adaptive streaming techniques have received significant research attention for omnidirectional videos [5,40,56] and point clouds [12,30,39,41]. However, further study is required for live real-time human point cloud reconstructions.…”