Temporal Context Mining for Learned Video Compression

Abstract

We address end-to-end learned video compression with a special focus on better learning and utilizing temporal contexts. For temporal context mining, we propose to store not only the previously reconstructed frames, but also the propagated features into the generalized decoded picture buffer. From the stored propagated features, we propose to learn multi-scale temporal contexts, and re-fill the learned temporal contexts into the modules of our compression scheme, including the contextual encoder-decoder, the frame generator, and the temporal context encoder. Our scheme discards the parallelization-unfriendly auto-regressive entropy model to pursue a more practical decoding time. We compare our scheme with x264 and x265 (representing industrial software for H.264 and H.265, respectively) as well as the official reference software for H. 264, H.265, and H.266 (JM, HM, and VTM, respectively). When intra period is 32 and oriented to PSNR, our scheme outperforms H.265-HM by 14.4% bit rate saving; when oriented to MS-SSIM, our scheme outperforms H.266-VTM by 21.1% bit rate saving.

Neural Video Compression with Diverse Contexts

“…Li et al proposed learning feature domain contexts as condition. Its following works [29,50] adopt feature propagation to boost performance.…”

“…The supplementary materials show the results using BT.601. In addition, we follow [29,50] and also test HEVC RGB dataset [15] when testing RGB videos, and there is no format change as HEVC RGB dataset itself is in RGB format.…”

“…Test Conditions. We follow [29,50] and test 96 frames for each video, and the intra period is set as 32. The low delay encoding setting is used, as the same with most existing works [1,28,35].…”

“…BD-Rate [5] In particular, we also compare with ECM [14] which is the prototype of next generation traditional codec. For the codec setting, we follow [29,50] and further use 10-bit as the intermediate representation when testing RGB, which leads to better compression ratio for the three traditional codecs. The detailed settings are shown in supplementary materials.…”

“…The detailed settings are shown in supplementary materials. As for the NVC benchmarks, we compare with the recent SOTA models including CANF-VC [21], Sheng 2021 [50], and DCVC MM22 [29].…”