Understanding Multi-Head Attention in Abstractive Summarization

Joris, Baan,; Hoeve, Maartje ter; Wees, Marlies van der; Schuth, Anne; Rijke, Maarten de

doi:10.48550/arxiv.1911.03898

Cited by 4 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The module dynamically adjusted the degree of attention to different positions in the input sequence by weighting the attention weights. When dealing with application categories with imbalanced sample sizes, this characteristic of MHSA allowed the model to focus on key features of the categories with small sample sizes, aiding in capturing differential information among different application category traffic data and improving the ability to recognize categories with small sample sizes [28].…”

Section: Multi-head Self-attention Modulementioning

confidence: 99%

DarkGuardNet: A deep learning framework for imbalanced dark web traffic identification and application classification

Niu,

Li,

Liu

2024

Preprint

View full text Add to dashboard Cite

The dark web was often utilized for illicit activities, data breaches, and the dissemination of malicious software. Researchers consistently employed various machine learning and deep learning approaches to detect dark web traffic. However, existing studies overlooked the comprehensive capture of multi-scale information in traffic data, resulting in an inability to fully extract features when dealing with complex structural data, especially in datasets with an imbalanced number of samples. To address this problem, our paper proposed DarkGuardNet for the recognition of dark web traffic and application type classification. Specifically, we conducted dark web traffic analysis based on sessions and designed a Spatio-temporal Feature Fusion (STFF) module to capture multi-scale feature correlations. This module extended the receptive field to deepen the understanding of complex data, allowing for the precise extraction of spatiotemporal features in imbalanced samples. In addition, we used Multi-Head Self-Attention (MHSA) to mine potential relationships between statistical features of dark web traffic, enabling the model to focus on key features of categories with small sample sizes. Finally, we conducted experiments on a new imbalanced dark web traffic dataset, formed by merging ISCXVPN and ISCXTor. The results indicated that the method achieved an accuracy of 0.999 in dark web traffic recognition and an accuracy of 0.986 in application type classification, surpassing other advanced methods. The Data is available at:https://github.com/niu954325618/Darknet2024/tree/main.

show abstract

Section: Multi-head Self-attention Modulementioning

confidence: 99%

DarkGuardNet: A deep learning framework for imbalanced dark web traffic identification and application classification

Niu,

Li,

Liu

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, the attention mechanism has helped solve many research problems [ 40 ]. This mechanism is a core part of the transformer architecture introduced by Vaswani et al, 2017 [ 41 ]. Multi-Head Attention is essential because, in simple self-attention, the model learns only one set of attention weights, which can be limiting.…”

Section: Preliminariesmentioning

confidence: 99%

TimeTector: A Twin-Branch Approach for Unsupervised Anomaly Detection in Livestock Sensor Noisy Data (TT-TBAD)

Kakar,

Hussain,

Kim

et al. 2024

Sensors

View full text Add to dashboard Cite

Unsupervised anomaly detection in multivariate time series sensor data is a complex task with diverse applications in different domains such as livestock farming and agriculture (LF&A), the Internet of Things (IoT), and human activity recognition (HAR). Advanced machine learning techniques are necessary to detect multi-sensor time series data anomalies. The primary focus of this research is to develop state-of-the-art machine learning methods for detecting anomalies in multi-sensor data. Time series sensors frequently produce multi-sensor data with anomalies, which makes it difficult to establish standard patterns that can capture spatial and temporal correlations. Our innovative approach enables the accurate identification of normal, abnormal, and noisy patterns, thus minimizing the risk of misinterpreting models when dealing with mixed noisy data during training. This can potentially result in the model deriving incorrect conclusions. To address these challenges, we propose a novel approach called “TimeTector-Twin-Branch Shared LSTM Autoencoder” which incorporates several Multi-Head Attention mechanisms. Additionally, our system now incorporates the Twin-Branch method which facilitates the simultaneous execution of multiple tasks, such as data reconstruction and prediction error, allowing for efficient multi-task learning. We also compare our proposed model to several benchmark anomaly detection models using our dataset, and the results show less error (MSE, MAE, and RMSE) in reconstruction and higher accuracy scores (precision, recall, and F1) against the baseline models, demonstrating that our approach outperforms these existing models.

show abstract

“…Most existing big models adopt deep Transformer as the basic architecture, and inevitably meet the over-parameterization problem. Early analysis on machine translation [476], abstractive summarization [477], and language understanding [478] have shown that a well-trained Transformer usually has redundant parameters, and can remove part of parameters without loss of performance. Recently, a series of work also discuss the over-parameterization problem in big models, containing the redundant heads in the multi-head attention layers [445,478], the sparse activation phenomenon in feed-forward network layer [479], and parameter redundancy problem of the whole Transformer [480,481].…”

Section: Model Analysismentioning

confidence: 99%

A Roadmap for Big Model

Yuan¹,

Zhao²,

Jiahong³

et al. 2022

Preprint

View full text Add to dashboard Cite

domains indexed by Google News. It contains 31 million documents with an average length of 793 BPE tokens. Like C4, it excludes examples with duplicate URLs. News dumps from December 2016 through March 2019 were used as training data, articles published in April 2019 from the April 2019 dump were used for evaluation. OpenWebText2(OWT2). OWT2 is an enhanced version of the original OpenWebTextCorpus, including content from multiple languages, document metadata, multiple dataset versions, and open source replication code, covering all Reddit submissions from 2005 up until April 2020. PubMed Central(PMC). PMC is a free full-text archive of biomedical and life sciences journal literature from the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM). The dataset is updated daily. In addition to full-text articles, they contain corrections, retractions, and expressions of concern, as well as file lists that include metadata for articles in each dataset.PMC obtained by open registration in Amazon Web Services (AWS) includes The PMC Open Access Subset and The Author Manuscript Dataset. The PMC Open Access Subset includes all articles and preprints in PMC with a machine-readable Creative Commons license that allows reuse. The Author Manuscript Dataset includes accepted author manuscripts collected under a funder policy in PMC and made available in machine-readable formats for text mining. ArXiv. ArXiv is a repository of 1.7 million articles, with relevant features such as article titles, authors, categories, abstracts, full text PDFs, and more. It provides open access to academic articles, covering many subdisciplines from vast branches of physics to computer science to everything in between, including math, statistics, electrical engineering, quantitative biology, and economics, which is helpful to the potential downstream applications of the research field. In addition, the writing language of LaTeX also contributes to the study of language models. Colossal Clean Crawled Corpus(C4). C4 is a colossal, cleaned version of Common Crawl's web crawl corpus. It is based on Common Crawl dataset and was used to train the T5 text-to-text Transformer models. The cleaned English version of C4 has 364,868,901 training examples and 364,608 validation examples, while the uncleaned English version has 1,063,805,324 training examples and 1,065,029 validation examples; the realnewslike version has 13,799,838 training examples and 13,863 validation examples, while the webtextlike version has 4,500,788 training examples and 4,493 validation examples. Wiki-40B. Wikipedia (Wiki-40B) is a clean-up text collection containing more than 40 Wikipedia language editions of pages corresponding to entities. The dataset is split into train/validation/test sets for each language. The training set has 2,926,536 examples, the validation set has 163,597 examples, and the test set has 162,274 examples. Wiki-40B is cleaned by a page filter to remove ambiguous, redirected, deleted, and non-physical pages. CLUECorpus2020. CLUECorpus2020 ...

show abstract

Understanding Multi-Head Attention in Abstractive Summarization

Cited by 4 publications

References 0 publications

DarkGuardNet: A deep learning framework for imbalanced dark web traffic identification and application classification

DarkGuardNet: A deep learning framework for imbalanced dark web traffic identification and application classification

TimeTector: A Twin-Branch Approach for Unsupervised Anomaly Detection in Livestock Sensor Noisy Data (TT-TBAD)

A Roadmap for Big Model

Contact Info

Product

Resources

About