BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies

Verelst, Thomas; Tuytelaars, Tinne

doi:10.1109/iccv48922.2021.00511

Cited by 11 publications

(2 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To be able to process UHD images in real-time and get good results, Zhang et al [40] propose a new Vision Transformer (ViT) architecture Multi-Scale Vision Longformer which provides image encodings with manageable computational cost. In addition, Thomas Verelst et al propose [32] BlockCopy operating on the selected region and reduce the number of calculations.…”

Section: Processing Methods Of Uhd Imagesmentioning

confidence: 99%

Single UHD Image Dehazing via Interpretable Pyramid Network

Xiao¹,

Zheng²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Currently, most single image dehazing models cannot run an ultra-high-resolution (UHD) image with a single GPU shader in real-time. To address the problem, we introduce the principle of infinite approximation of Taylor's theorem with the Laplace pyramid pattern to build a model which is capable of handling 4K hazy images in real-time.The N branch networks of the pyramid network correspond to the N constraint terms in Taylor's theorem. Low-order polynomials reconstruct the low-frequency information of the image (e.g. color, illumination). High-order polynomials regress the high-frequency information of the image (e.g. texture). In addition, we propose a Tucker reconstructionbased regularization term that acts on each branch network of the pyramid model. It further constrains the generation of anomalous signals in the feature space. Extensive experimental results demonstrate that our approach can not only run 4K images with haze in real-time on a single GPU (80FPS) but also has unparalleled interpretability. The developed method achieves state-of-the-art (SOTA) performance on two benchmarks (O/I-HAZE) and our updated 4KID dataset while providing the reliable groundwork for subsequent optimization schemes.

show abstract

Section: Processing Methods Of Uhd Imagesmentioning

confidence: 99%

Single UHD Image Dehazing via Interpretable Pyramid Network

Xiao¹,

Zheng²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…PASS first filters out trivial patches according to the temporal redundancy of input, then skips the corresponding matrix multiplication to achieve less MAC count and time cost. In practice, we can leverage the Block-wise Sparse Convolution (Ren et al 2018;Verelst and Tuytelaars 2021) technique to handle the engineering implementation.…”

Section: Preliminarymentioning

confidence: 99%

PASS: Patch Automatic Skip Scheme for Efficient Real-Time Video Perception on Edge Devices

Zhou

Guo

Pan

et al. 2023

AAAI

View full text Add to dashboard Cite

Real-time video perception tasks are often challenging over the resource-constrained edge devices due to the concerns of accuracy drop and hardware overhead, where saving computations is the key to performance improvement. Existing methods either rely on domain-specific neural chips or priorly searched models, which require specialized optimization according to different task properties. In this work, we propose a general and task-independent Patch Automatic Skip Scheme (PASS), a novel end-to-end learning pipeline to support diverse video perception settings by decoupling acceleration and tasks. The gist is to capture the temporal similarity across video frames and skip the redundant computations at patch level, where the patch is a non-overlapping square block in visual. PASS equips each convolution layer with a learnable gate to selectively determine which patches could be safely skipped without degrading model accuracy. As to each layer, a desired gate needs to make flexible skip decisions based on intermediate features without any annotations, which cannot be achieved by conventional supervised learning paradigm. To address this challenge, we are the first to construct a tough self-supervisory procedure for optimizing these gates, which learns to extract contrastive representation, i.e., distinguishing similarity and difference, from frame sequence. These high-capacity gates can serve as a plug-and-play module for convolutional neural network (CNN) backbones to implement patch-skippable architectures, and automatically generate proper skip strategy to accelerate different video-based downstream tasks, e.g., outperforming the state-of-the-art MobileHumanPose (MHP) in 3D pose estimation and FairMOT in multiple object tracking, by up to 9.43 times and 12.19 times speedups, respectively. By directly processing the raw data of frames, PASS can generalize to real-time video streams on commodity edge devices, e.g., NVIDIA Jetson Nano, with efficient performance in realistic deployment.

show abstract

MVSparse: Distributed Cooperative Multi-camera Multi-target Tracking on the Edge

Nalaie,

Zheng

2024

2024 IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)

View full text Add to dashboard Cite

BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies

Cited by 11 publications

References 33 publications

Single UHD Image Dehazing via Interpretable Pyramid Network

Single UHD Image Dehazing via Interpretable Pyramid Network

PASS: Patch Automatic Skip Scheme for Efficient Real-Time Video Perception on Edge Devices

MVSparse: Distributed Cooperative Multi-camera Multi-target Tracking on the Edge

Contact Info

Product

Resources

About