Coarse-to-Fine Embedded PatchMatch and Multi-Scale Dynamic Aggregation for Reference-based Super-Resolution

Abstract

Reference-based super-resolution (RefSR) has made significant progress in producing realistic textures using an external reference (Ref) image. However, existing RefSR methods obtain high-quality correspondence matchings consuming quadratic computation resources with respect to the input size, limiting its application. Moreover, these approaches usually suffer from scale misalignments between the lowresolution (LR) image and Ref image. In this paper, we propose an Accelerated Multi-Scale Aggregation network (AMSA) for Reference-based Super-Resolution, including Coarse-to-Fine Embedded PatchMatch (CFE-PatchMatch) and Multi-Scale Dynamic Aggregation (MSDA) module. To improve matching efficiency, we design a novel Embedded PatchMacth scheme with random samples propagation, which involves end-to-end training with asymptotic linear computational cost to the input size. To further reduce computational cost and speed up convergence, we apply the coarseto-fine strategy on Embedded PatchMacth constituting CFE-PatchMatch. To fully leverage reference information across multiple scales and enhance robustness to scale misalignment, we develop the MSDA module consisting of Dynamic Aggregation and Multi-Scale Aggregation. The Dynamic Aggregation corrects minor scale misalignment by dynamically aggregating features, and the Multi-Scale Aggregation brings robustness to large scale misalignment by fusing multi-scale information. Experimental results show that the proposed AMSA achieves superior performance over state-of-the-art approaches on both quantitative and qualitative evaluations.

Any-resolution Training for High-resolution Image Synthesis

“…Furthermore, though generations continue to improve, there remains a persistent domain shift between synthesized and real images [60,61]. Finally, super-resolution is a local, conditional problem where the global structure is dictated by the low-resolution input, and optionally an additional highresolution reference image [62,63,64,65,66]. We synthesize plausible images unconditionally, leveraging a set of high-resolution images to produce realistic global structure and coherent ne details.…”

Dual Camera Based High Spatio-Temporal Resolution Video Generation For Wide Area Surveillance

“…The literature presents various RefSR methods that rely on patch matching between the reference and lowresolution images, followed by the transfer of textures from the REF to the LR image [11,13,14,15,19,20,21,22]. Despite achieving competitive results, these methods face challenges in capturing fine-grained information between successive frames in a video feed.…”

“…Nevertheless, [19] faces performance degradation when dealing with substantial scale misalignments between the REF and LR images. [14,21] offer an alternative approach to the problem by introducing coarse-to-fine correspondence matching instead of exhaustively iterating through all potential matches, which serves to reduce model complexity while achieving effective results. [16] divides the RefSR task into two distinct stages.…”

“…This reference image enhances the superresolution process by facilitating texture transfer between high-quality spatial features and their low-quality counterparts in the LR image. RefSR consistently produces quantitatively and visually better results when compared to SISR, mainly due to the effective transfer of relevant textures from the REF image to the LR image [2,11,12,13,14,15,16,18,19,20,21,22]. RefSRbased networks typically operate in the spatial domain, employing a correspondence matching strategy between all possible pairs of REF and LR patches for subsequent texture transfer.…”

“…These mechanisms encompass both hard and softattention techniques, facilitating the matching of similar patches while assigning weights to these correspondences to balance their influence based on patch similarity. Despite its effectiveness, this approach requires iterating through all possible patch pairs, which results in quadratic computational complexity [21]. Hence, there is an imperative need for an efficient method that optimizes both memory usage and execution time, allowing to limit the computational complexity and yet exploit the features generated by attention mechanisms to their fullest extent.…”