Differentiable Window for Dynamic Local Attention

Nguyen, Thi Hoa; Nguyen, Xuan-Phi; Joty, Shafiq; Li, Xiaoli

doi:10.48550/arxiv.2006.13561

Cited by 2 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Longformer [1] utilizes dilated sliding window attention to combine local and global information. And [25] enables more focused attentions by dynamic differentiable windows.…”

Section: Variable Attentionmentioning

confidence: 99%

Boosting Crowd Counting via Multifaceted Attention

Lin¹,

Ma²,

Ji³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper focuses on the challenging crowd counting task. As large-scale variations often exist within crowd images, neither fixed-size convolution kernel of CNN nor fixed-size attention of recent vision transformers can well handle this kind of variations. To address this problem, we propose a Multifaceted Attention Network (MAN) to improve transformer models in local spatial relation encoding. MAN incorporates global attention from vanilla transformer, learnable local attention, and instance attention into a counting model. Firstly, the local Learnable Region Attention (LRA) is proposed to assign attention exclusive for each feature location dynamically. Secondly, we design the Local Attention Regularization to supervise the training of LRA by minimizing the deviation among the attention for different feature locations. Finally, we provide an Instance Attention mechanism to focus on the most important instances dynamically during training. Extensive experiments on four challenging crowd counting datasets namely ShanghaiTech, UCF-QNRF, JHU++, and NWPU have validated the proposed method. Code: https://github.com/LoraLinH/Boosting-Crowd-Counting-via-Multifaceted-Attention.

show abstract

“…Longformer [1] utilizes dilated sliding window attention to combine local and global information. And [25] enables more focused attentions by dynamic differentiable windows.…”

Section: Variable Attentionmentioning

confidence: 99%

Boosting Crowd Counting via Multifaceted Attention

Lin¹,

Ma²,

Ji³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…for the local attention. However, in [6], they utilize a differential window instead of a Gaussian mask. For that reason, we transfer the idea of their fusion process to our approach:…”

Section: Bias Attention Fusionmentioning

confidence: 99%

“…Since the improvement mentioned in [6,20] is located in the domain of machine translation and language modeling, it is not clear if the same application holds for the local attention in ASR. Therefore, we perform a short ablation study where to apply localness and to identify the most effective way to fuse the global score from Equation 10 and local score from Equation 11.…”

Section: Ablation Studymentioning

confidence: 99%

“…Evaluation was done on TEDLIUMv2 [9] and was decoded with a beam size of 40. Location of Localness Recent approaches as [6,20] already demonstrated that the location of the local attention in the encoder's SA of the model is relevant and improves the transformer performance. They argue that the improvement results in the fact that the lower layers of the model process more low-level features, which contain more local information.…”

Section: Ablation Studymentioning

confidence: 99%

“…Recently, Nguyen et al [6] closed this gap by proposing a fully differentiable window, which is also applicable to the SA in the encoder, and the decoder, and in the cross-attention between encoder and decoder. They investigated different ways of adding the local window to the SA mechanism and where the local mask had the most significant impact on the performance of the overall transformer model.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Induced Local Attention for Transformer Models in Speech Recognition

Watzel

Kürzinger

et al. 2021

Speech and Computer

View full text Add to dashboard Cite

The transformer models and their variations currently are considered the prime model architectures in speech recognition since they yield state-of-the-art results on several datasets. Their main strength lies in the self-attention mechanism, where the models receive the ability to calculate a score over the whole input sequence and focus on essential aspects of the sequence. However, the attention score has some flaws. It is heavily global-dependent since it takes the whole sequence into account and normalizes along the sequence length. Our work presents a novel approach for a dynamic fusion between the global and a local attention score based on a Gaussian mask. The small networks for learning the fusion process and the Gaussian masks require only few additional parameters and are simple to add to current transformer architectures. With our exhaustive evaluation, we determine the effect of localness in the encoder layers and examine the most effective fusion approach. The results on the dataset TEDLIUMv2 demonstrate a steady improvement on the dev and the test set for the base transformer model equipped with our proposed fusion procedure for local attention.

show abstract

Differentiable Window for Dynamic Local Attention

Cited by 2 publications

References 11 publications

Boosting Crowd Counting via Multifaceted Attention

Boosting Crowd Counting via Multifaceted Attention

Induced Local Attention for Transformer Models in Speech Recognition

Contact Info

Product

Resources

About