An Attentive Survey of Attention Models

Chaudhari, Sneha; Mithal, Varun; Polatkan, Gungor; Ramanath, Rohan

doi:10.48550/arxiv.1904.02874

Cited by 47 publications

(68 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many attention-based models have been developed to improve the performance of classification and regression algorithms. Surveys of various attention-based models are available in [2,3,4,5,17,1].…”

Section: Related Workmentioning

confidence: 99%

“…It can be viewed as a learnable mask which emphasizes relevant information in a feature map. It is pointed out in [2,9] that the original idea of attention can be understood from the statistics point of view applying the Nadaraya-Watson kernel regression model [10,11].…”

Section: Preliminary: the Attention Mechanismmentioning

confidence: 99%

“…Many attention-based models are also applied to the computer vision area, including image-based analysis, visual question answering, etc. Detailed surveys and reviews of attention, its forms, properties and applications can be found in [2,3,4,5,1].…”

Section: Introductionmentioning

confidence: 99%

“…Many neural attention models are simple, scalable, flexible, and with promising results in several application domains [3]. However, attention is considered as an essential component of neural architectures [2], the attention weights are learned by incorporating an additional feed forward neural network within the architectures. This implies that they have all problems which are encountered in neural networks, including, overfitting, many tuning parameters, requirements of a large amount of data, the black-box nature, expensive computations.…”

Section: Introductionmentioning

confidence: 99%

“…It is pointed out in [2,9] that the original idea of attention can be understood from the statistics point of view applying the Nadaraya-Watson kernel regression model [10,11]. According to the Nadaraya-Watson regression, weights or normalized kernels conform with relevance of a training instance to a target feature vector.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Attention-based Random Forest and Contamination Model

Utkin¹,

Konstantinov²

2022

Preprint

View full text Add to dashboard Cite

A new approach called ABRF (the attention-based random forest) and its modifications for applying the attention mechanism to the random forest (RF) for regression and classification are proposed. The main idea behind the proposed ABRF models is to assign attention weights with trainable parameters to decision trees in a specific way. The weights depend on the distance between an instance, which falls into a corresponding leaf of a tree, and instances, which fall in the same leaf. This idea stems from representation of the Nadaraya-Watson kernel regression in the form of a RF. Three modifications of the general approach are proposed. The first one is based on applying the Huber's contamination model and on computing the attention weights by solving quadratic or linear optimization problems. The second and the third modifications use the gradient-based algorithms for computing trainable parameters. Numerical experiments with various regression and classification datasets illustrate the proposed method.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Preliminary: the Attention Mechanismmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Attention-based Random Forest and Contamination Model

Utkin¹,

Konstantinov²

2022

Preprint

View full text Add to dashboard Cite

show abstract

An effective framework for semistructured document classification via hierarchical attention model

Zhao

Fang

Zhang

et al. 2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

Recent years have witnessed the rapidly growing of the amount of semistructured documents in real‐world applications. Due to the huge size of the real‐world data, how to manage semistructured documents effectively is a big challenge for researchers. As a fundamental task in natural language processing field, document classification is a feasible way to handle the large‐scale semistructured documents. However, existing methods fail to explicitly take advantage of the hierarchical semantics in semistructured documents. It's known that the contained semantics is beneficial for understanding the semistructured documents. Considering the hierarchical structure of a given semistructured document, we propose a semistructured document classification framework which explicitly utilizes the semantic hierarchical attention mechanism. More specifically, the hierarchical attention mechanism and graph neural network are employed to model semistructured documents, by which the multilevel semantic relationships and grammatical information are considered. Moreover, we propose an adaptive class cost learning method to treat the issue of data imbalance. Comprehensive experiments are conducted on two real‐world data sets, and the results demonstrate that our framework performs better than selected baselines for semistructured document classification.

show abstract

Image denoising in the deep learning era

2022

View full text Add to dashboard Cite

Over the last decade, the number of digital images captured per day witnessed a massive explosion. Nevertheless, the visual quality of photographs is often degraded by noise during image acquisition or transmission. With the re-emergence of deep neural networks, the performance of image denoising techniques has been substantially improved in recent years. The objective of this paper is to provide a comprehensive survey of recent advances in image denoising techniques based on deep neural networks. In doing so, we commence with a thorough description of the fundamental preliminaries of the image denoising problem followed by highlighting the benchmark datasets and the widely used metrics for objective assessments. Subsequently, we study the existing deep denoisers in the supervised and unsupervised categories and review the technical specifics of some representative methods within each category. Last but not least, we conclude the analysis by remarking on trends and challenges in the development of better state-of-the-art algorithms and future research.

show abstract

An Attentive Survey of Attention Models

Cited by 47 publications

References 0 publications

Attention-based Random Forest and Contamination Model

Attention-based Random Forest and Contamination Model

An effective framework for semistructured document classification via hierarchical attention model

Image denoising in the deep learning era

Contact Info

Product

Resources

About