Non‐local neural networks combined with local importance‐based pooling for space‐time video super‐resolution

Abstract

Compared with convolutional operation, non‐local operation can directly capture long‐range dependencies and thus has a larger receptive field. However, the computation and memory consumption of non‐local operation is much higher than convolutional operation, so it cannot be used repeatedly as a general component directly. In this paper, in order to balance the accuracy and computational complexity of non‐local enhancement, the non‐local operation is simplified based on local importance‐based pooling, which can dynamically extract discriminative features during the down‐sampling process by learning adaptive weights. Such simplified non‐local enhancement is able to prevent unacceptable computational consumption caused by directly processing the entire feature maps containing a large number of features. In order to verify the effectiveness of the proposed method, 2D and 3D feature extraction blocks are constructed based on the simplified non‐local operations, and they are stacked as feature extraction networks for space‐time video super‐resolution task, which aims to increase resolution in both time and space simultaneously. Extensive experiments demonstrate that the proposed simplified non‐local networks can effectively improve the performance of space‐time video super‐resolution task both quantitatively and qualitatively.

Image stitching based on deep learning

“…The choice of the paired function f form is Embedded Gaussian [7]: The warped image is first downsampled to a low resolution, defined as 256 × 256. convolution of different sizes is applied to the LR image for multi-scale fusion convolutional feature extraction to obtain useful information from the original image. In addition, skip-join is used to connect low-level and high-level features with the same resolution [8] .The figure 5 shows the feature map of the low-resolution deformation branch.…”

“…Using the all-ones matrixE instead of I , the I , the seam mask can be calculated using equations. (6) (7) and equation (8).…”