BiFormer: Vision Transformer with Bi-Level Routing Attention

Abstract

As the core building block of vision transformers, attention is a powerful tool to capture long-range dependency. However, such power comes at a cost: it incurs a huge computation burden and heavy memory footprint as pairwise token interaction across all spatial locations is computed. A series of works attempt to alleviate this problem by introducing handcrafted and content-agnostic sparsity into attention, such as restricting the attention operation to be inside local windows, axial stripes, or dilated windows. In contrast to these approaches, we propose a novel dynamic sparse attention via bi-level routing to enable a more flexible allocation of computations with content awareness. Specifically, for a query, irrelevant key-value pairs are first filtered out at a coarse region level, and then fine-grained token-to-token attention is applied in the union of remaining candidate regions (i.e., routed regions). We provide a simple yet effective implementation of the proposed bilevel routing attention, which utilizes the sparsity to save both computation and memory while involving only GPUfriendly dense matrix multiplications. Built with the proposed bi-level routing attention, a new general vision transformer, named BiFormer, is then presented. As BiFormer attends to a small subset of relevant tokens in a query adaptive manner without distraction from other irrelevant ones, it enjoys both good performance and high computational efficiency, especially in dense prediction tasks. Empirical results across several computer vision tasks such as image classification, object detection, and semantic segmentation verify the effectiveness of our design. Code is available at https://github.com/rayleizhu/BiFormer.

A microfluidic microalgae detection system for cellular physiological response based on an object detection algorithm

“…Concerning the model architecture, the integration of the bilevel routing attention (BiLRA) 47 and a new detection head was effected to amplify the detection model's performance concerning diminutive cell entities and the structure of BiLRA is shown in Fig. S3.…”

DB-EAC and LSTR: DBnet based seal text detection and Lightweight Seal Text Recognition

“…In this paper, inspired by the application of Bifomer [ 27 ] in vision, a two-layer routing attention mechanism is utilized to filter out most of the irrelevant K-V pairs in the coarse-grained region, followed by applying token-to-token attention to focus on a small portion of the relevant tokens, which provides good performance and computational efficiency since it does not distract the attention of irrelevant tokens. We propose to use stacked multi-head self-attention to construct an inference layer.…”

Improved algorithm for small target detection in remote sensing images based on YOLOv5s

“…To improve the detection performance of small targets in remote sensing images, this paper proposes an improved algorithm based on YOLOv5s-6.0, called YOLOv5s-RSD. The main improvements are summarized in three points: (1) using SPD-Conv [14] to replace the original downsampling method to reduce the loss of detailed information in the downsampling process; (2) adding a small target detection head to fully utilize small target features, and recalculating the optimal pre-selected box using K-means; (3) introducing Biformer [15] attention to enhance the focus on useful features.…”