Render2MPEG: A Perception‐based Framework Towards Integrating Rendering and Video Compression

Abstract

Currently 3D animation rendering and video compression are completely independent processes even if rendered frames are streamed on-the-fly within a client-server platform. In such scenario, which may involve time-varying transmission bandwidths and different display characteristics at the client side, dynamic adjustment of the rendering quality to such requirements can lead to a better use of server resources. In this work, we present a framework where the renderer and MPEG codec are coupled through a straightforward interface that provides precise motion vectors from the rendering side to the codec and perceptual error thresholds for each pixel in the opposite direction. The perceptual error thresholds take into account bandwidth-dependent quantization errors resulting from the lossy compression as well as image content-dependent luminance and spatial contrast masking. The availability of the discrete cosine transform (DCT) coefficients at the codec side enables to use advanced models of the human visual system (HVS) in the perceptual error threshold derivation without incurring any significant cost. Those error thresholds are then used to control the rendering quality and make it well aligned with the compressed stream quality. In our prototype system we use the lightcuts technique developed by Walter et al., which we enhance to handle dynamic image sequences, and an MPEG-2 implementation. Our results clearly demonstrate many advantages of coupling the rendering with video compression in terms of faster rendering. Furthermore, temporally coherent rendering leads to a reduction of temporal artifacts

A Survey of Interactive Remote Rendering Systems

“…The hybrid video coders [Wang et al 2001] integrate lossy image compression tools with motioncompensation tools to exploit the temporal redundancy. The hybrid video coders have been standardized as, for example, MPEG-4 [Schafer 1998] and H.264/AVC [Wiegand et al 2003], which (or their variations) are used by several modern remote rendering systems [Noimark and Cohen-Or 2003;Lamberti and Sanna 2007;Jurgelionis et al 2009;De Winter et al 2006;Perlman et al 2010;Shi et al 2011a;Herzog et al 2008;Huang et al 2013]. The video coding tools proposed in Tran et al [1998], Bayazit [1999], and Liu et al [2007] adopted some or all of the real-time coding constraints discussed in Reddy and Chunduri [2006] and Schreier et al [2006].…”

Space-time volumetric depth images for in-situ visualization

“…To maintain interactivity, the transfer size is typically reduced by employing JPEG [6] or MPEG [5] compression. An alternative technique employing a CUDA-based parallel compression method was presented by Lietsch and Marquardt [9], while Pajak et al [14] discuss efficient compression and streaming of frames rendered from a dynamic 3D model.…”

Stochastic Progressive Photon Mapping for Dynamic Scenes

“…This is used for video compression, eg. MPEG [HKMS08]. The main difference to our method is that we do not display a visually acceptable approximation for selected in-between frames, but compute a correct simulation in each frame.…”