Video super-resolution by generative adversarial network with 3D convolutional neural networks

Abstract

For effective super-resolution processing of video images, video images should not be processed as frame-by-frame twodimensional information, but as spatio-temporal information, including information in the time axis direction. Most of the proposed video super-resolution processing based on deep learning uses 2D convolutional neural networks (CNNs). Therefore, the system is based on the 2D CNNs with the additional processing related to change from frame to frame. Instead of processing video images separately in space and time, comprehensive processing as spatio-temporal information can be expected to be more flexible and effective. In this research, we propose a video super-resolution process that processes spatio-temporal information by 3D-CNNs and GAN (Generative adversarial networks). By using 3D-CNNs, the configuration does not require motion alignment as preprocessing. Since the essential purpose of the super-resolution process is to predict missing high-frequency components, we added a process that directly predicts the difference between the high-resolution image and the corresponding bicubic interpolated low-resolution image.