A General Survey on Attention Mechanisms in Deep Learning

Brauwers, Gianni; Frăsincar, Flavius

doi:10.1109/tkde.2021.3126456

Cited by 220 publications

(90 citation statements)

References 87 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Consider a scoring function f defined: 𝑓: ℝ × ℝ ↦ ℝ, which determines the assigned weight to its vector and indicates the important vectors. Therefore, the context vector (𝑐 ) is a weighted sum of hidden states H = {ℎ , ℎ , … , ℎ } and represents the important information for the current time step as depicted in the following equations [52]: Consider a scoring function f defined: f : R m × R m → R , which determines the assigned weight to its vector and indicates the important vectors. Therefore, the context vector (c t ) is a weighted sum of hidden states H = {h 1 , h 2 , .…”

Section: Attention Layersmentioning

confidence: 99%

“…. , h t−1 } and represents the important information for the current time step as depicted in the following equations [52]:…”

Section: Attention Layersmentioning

confidence: 99%

“…The context vector is then unified with the previous hidden state unit S t−1 and previous target output y t−1 to predict the current hidden state S t as depicted in the following equation [52]:…”

Section: Attention Layersmentioning

confidence: 99%

See 2 more Smart Citations

Attention-Based Distributed Deep Learning Model for Air Quality Forecasting

et al. 2022

View full text Add to dashboard Cite

Air quality forecasting has become an essential factor in facilitating sustainable development worldwide. Several countries have implemented monitoring stations to collect air pollution particle data and meteorological information using parameters such as hourly timespans. This research focuses on unravelling a new framework for air quality prediction worldwide and features Busan, South Korea as its model city. The paper proposes the application of an attention-based convolutional BiLSTM autoencoder model. The proposed deep learning model has been trained on a distributed framework, referred to data parallelism, to forecast the intensity of particle pollution ( and ). The algorithm automatically learns the intrinsic correlation among the particle pollution in different locations. Each location’s meteorological and traffic data is extensively exploited to improve the model’s performance. The model has been trained using air quality particle data and car traffic information. The traffic information is obtained by a device which counts cars passing a specific area through the YOLO algorithm, and then sends the data to a stacked deep autoencoder to be encoded alongside the meteorological data before the final prediction. In addition, multiple one-dimensional CNN layers are used to obtain the local spatial features jointly with a stacked attention-based BiLSTM layer to figure out how air quality particles are correlated in space and time. The evaluation of the new attention-based convolutional BiLSTM autoencoder model was derived from data collected and retrieved from comprehensive experiments conducted in South Korea. The results not only show that the framework outperforms the previous models both on short- and long-term predictions but also indicate that traffic information can improve the accuracy of air quality forecasting. For instance, during prediction, the proposed attention-based model obtained the lowest MAE (5.02 and 22.59, respectively, for short-term and long-term prediction), RMSE (7.48 and 28.02) and SMAPE (17.98 and 39.81) among all the models, which indicates strong accuracy between observed and predicted values. It was also found that the newly proposed model had the lowest average training time compared to the baseline algorithms. Furthermore, the proposed framework was successfully deployed in a cloud server in order to provide future air quality information in real time and when needed.

show abstract

Section: Attention Layersmentioning

confidence: 99%

“…. , h t−1 } and represents the important information for the current time step as depicted in the following equations [52]:…”

Section: Attention Layersmentioning

confidence: 99%

See 1 more Smart Citation

Attention-Based Distributed Deep Learning Model for Air Quality Forecasting

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Meanwhile, attention mechanism focuses on the important information of an image, which is more expandable and robust ( Brauwers and Frasincar, 2021 ) that improves the performance of CNN models. Different attention modules have been employed.…”

Section: Deep Learning For the Chinese Herbal Slices Image Recognitionmentioning

confidence: 99%

Combined Channel Attention and Spatial Attention Module Network for Chinese Herbal Slices Automated Recognition

Wang

et al. 2022

Front. Neurosci.

View full text Add to dashboard Cite

Chinese Herbal Slices (CHS) are critical components of Traditional Chinese Medicine (TCM); the accurate recognition of CHS is crucial for applying to medicine, production, and education. However, existing methods to recognize the CHS are mainly performed by experienced professionals, which may not meet vast CHS market demand due to time-consuming and the limited number of professionals. Although some automated CHS recognition approaches have been proposed, the performance still needs further improvement because they are primarily based on the traditional machine learning with hand-crafted features, resulting in relatively low accuracy. Additionally, few CHS datasets are available for research aimed at practical application. To comprehensively address these problems, we propose a combined channel attention and spatial attention module network (CCSM-Net) for efficiently recognizing CHS with 2-D images. The CCSM-Net integrates channel and spatial attentions, focusing on the most important information as well as the position of the information of CHS image. Especially, pairs of max-pooling and average pooling operations are used in the CA and SA module to aggregate the channel information of the feature map. Then, a dataset of 14,196 images with 182 categories of commonly used CHS is constructed. We evaluated our framework on the constructed dataset. Experimental results show that the proposed CCSM-Net indicates promising performance and outperforms other typical deep learning algorithms, achieving a recognition rate of 99.27%, a precision of 99.33%, a recall of 99.27%, and an F1-score of 99.26% with different numbers of CHS categories.

show abstract

“…The New York officers drove to Poughkeepsie , N.Y. , from Albany , then took a train to Grand Central Although great success has been achieved in previous work on extracting entities and relations, most relation extraction models ignore the fact that subjects contain rich semantic information on objects and relations that has immediate relevance to the relation extraction. Moreover, Wei et al [11] and Sun et al [12] have proven that fusing the relevant representations into context representation enables the enhancement of model performance in relation extraction, and the attention model has proven its effectiveness in representation fusion [13]. Furthermore, Lai et al [14] have utilized an attention model to extract relational triplets, further proving its effectiveness in relation extraction.…”

Section: Seomentioning

confidence: 99%

RECA: Relation Extraction Based on Cross-Attention Neural Network

Huang¹,

Guo²,

Zhang³

et al. 2022

Electronics

View full text Add to dashboard Cite

Extracting entities and relations, as a crucial part of many tasks in natural language processing, transforms the unstructured text information into structured information and provides corresponding data support for knowledge graph (KG) and knowledge vault (KV) construction. Nevertheless, the mainstream relation-extraction methods, the pipeline method and the joint method, ignore the dependency between the subject entity and the object entity. This work introduces a pre-trained BERT model and a dilated gated convolutional neural network (DGCNN) as an encoder to distinguish the long-range semantics representation from the input sequence. In addition, we propose a cross-attention neural network as a decoder to learn the importance of each subject word for each word of the input sequence. Experiments were undertaken with two extensive datasets, the New York Times Corpus (NYT) and WebNLG Corpus, and showed that our model performs significantly better than the CasRel model, outperforming the baseline by 1.9% and 0.7% absolute gain in terms of F1-score.

show abstract

A General Survey on Attention Mechanisms in Deep Learning

Cited by 220 publications

References 87 publications

Attention-Based Distributed Deep Learning Model for Air Quality Forecasting

Attention-Based Distributed Deep Learning Model for Air Quality Forecasting

Combined Channel Attention and Spatial Attention Module Network for Chinese Herbal Slices Automated Recognition

RECA: Relation Extraction Based on Cross-Attention Neural Network

Contact Info

Product

Resources

About