Video Summarization With Frame Index Vision Transformer

Hsu, Tien-Pen; Liao, Yi-Sheng; Huang, Chun-Rong

doi:10.23919/mva51890.2021.9511350

Cited by 5 publications

(1 citation statement)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Transformer has emerged promising potential in computer vision (Cai et al 2023;Hsu, Liao, and Huang 2023). IPT (Chen et al 2021) is a backbone model based on the standard transformer for various low-level tasks and is pre-trained on large-scale datasets with abundant computational resources.…”

Section: Efficient Transformer-based Sr Methodsmentioning

confidence: 99%

AdaFormer: Efficient Transformer with Adaptive Token Sparsification for Image Super-resolution

Luo,

Ai,

Liang

et al. 2024

AAAI

View full text Add to dashboard Cite

Efficient transformer-based models have made remarkable progress in image super-resolution (SR). Most of these works mainly design elaborate structures to accelerate the inference of the transformer, where all feature tokens are propagated equally. However, they ignore the underlying characteristic of image content, i.e., various image regions have distinct restoration difficulties, especially for large images (2K-8K), failing to achieve adaptive inference. In this work, we propose an adaptive token sparsification transformer (AdaFormer) to speed up the model inference for image SR. Specifically, a texture-relevant sparse attention block with parallel global and local branches is introduced, aiming to integrate informative tokens from the global view instead of only in fixed local windows. Then, an early-exit strategy is designed to progressively halt tokens according to the token importance. To estimate the plausibility of each token, we adopt a lightweight confidence estimator, which is constrained by an uncertainty-guided loss to obtain a binary halting mask about the tokens. Experiments on large images have illustrated that our proposal reduces nearly 90% latency against SwinIR on Test8K, while maintaining a comparable performance.

show abstract

Section: Efficient Transformer-based Sr Methodsmentioning

confidence: 99%