In-network view re-sampling for interactive free viewpoint video streaming

Abstract

Interactive free viewpoint video offers the possibility for each user to independently choose the views of a 3D scene to be displayed at the decoder. The visual content is commonly represented by N texture and depth map pairs that capture different viewpoints. A server selects an appropriate subset of M ≤ N views for transmission, so that the user can freely navigate in the corresponding window of viewpoints without being affected by network delay. During navigation, a user can synthesize any intermediate virtual view image in the navigation window via depth-image-based rendering (DIBR) using two nearby camera views as references. When the available bandwidth is too small to transmit all camera views typically used to synthesize views in the navigation window, we propose to synthesize intermediate virtual views as new references for transmission-a resampling of viewpoints for the 3D scene-so that the synthesized view distortion within the navigation window is minimized. We formulate a combinatorial optimization problem to find the best set of M virtual views to synthesize as new references, and show that the problem is NP-hard. We approximate the original problem with a new reference view equivalence model and derive in this case an optimal dynamic programming algorithm to determine the best set of M views to be transmitted to each user. Experimental results show that synthesizing virtual views as new references for client-side view synthesis can outperform simple selection from camera views by up to 0.73dB in synthesized view quality.

In-Network View Synthesis for Interactive Multiview Video Systems

“…In [48], another functional trend is empirically demonstrated, where a reference view v L with distortion D(v L ) was well approximated as a further-away equivalent…”

“…In [48], another functional trend is empirically demonstrated, where a reference view v L with distortion D(v L ) was well approximated as a further-away equivalent reference view just means that the equivalent reference pair for (v L , v R ) are closer to u than the equivalent reference pair for (v ′ L , v ′ R ). Using the same previous argument, we see that the equivalent reference pair for…”