Simplified Self-Attention for Transformer-Based end-to-end Speech Recognition

Luo, Haoneng; Zhang, Shiliang; Lei, Ming; Xie, Lei

doi:10.1109/slt48900.2021.9383581

Cited by 28 publications

(15 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The architecture was trained to perform both speech recognition and speech translation. Works such as those in [19][20][21] have investigated and proposed methods to improve the attention mechanism of the transformer for ASR. In [22], the authors conducted series of experiments to on RNNs against transformers for a number of ASR tasks and reported superior results with transformers.…”

Section: Existing Work On Transformers For Asrmentioning

confidence: 99%

A Bidirectional Context Embedding Transformer for Automatic Speech Recognition

et al. 2022

View full text Add to dashboard Cite

Transformers have become popular in building end-to-end automatic speech recognition (ASR) systems. However, transformer ASR systems are usually trained to give output sequences in the left-to-right order, disregarding the right-to-left context. Currently, the existing transformer-based ASR systems that employ two decoders for bidirectional decoding are complex in terms of computation and optimization. The existing ASR transformer with a single decoder for bidirectional decoding requires extra methods (such as a self-mask) to resolve the problem of information leakage in the attention mechanism This paper explores different options for the development of a speech transformer that utilizes a single decoder equipped with bidirectional context embedding (BCE) for bidirectional decoding. The decoding direction, which is set up at the input level, enables the model to attend to different directional contexts without extra decoders and also alleviates any information leakage. The effectiveness of this method was verified with a bidirectional beam search method that generates bidirectional output sequences and determines the best hypothesis according to the output score. We achieved a word error rate (WER) of 7.65%/18.97% on the clean/other LibriSpeech test set, outperforming the left-to-right decoding style in our work by 3.17%/3.47%. The results are also close to, or better than, other state-of-the-art end-to-end models.

show abstract

Section: Existing Work On Transformers For Asrmentioning

confidence: 99%

A Bidirectional Context Embedding Transformer for Automatic Speech Recognition

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Where all H, M , and W vectors have a size of 1024 (≈ denotes downstream operations for key/val). The above idea was inspired by another simplified attention implementation where they replaced the fully connected operations for Q, K and V with an elementwise multiplication with trainable parameters summed over time [8]. As can be seen in…”

Section: Alternate-attention Head Experimentsmentioning

confidence: 99%

SHAQ: Single Headed Attention with Quasi-Recurrence

Nashwin¹,

Kushner²,

Sangeet³

et al. 2021

Preprint

View full text Add to dashboard Cite

Natural Language Processing research has recently been dominated by large scale transformer models. Although they achieve state of the art on many important language tasks, transformers often require expensive compute resources, and days spanning to weeks to train. This is feasible for researchers at big tech companies and leading research universities, but not for scrappy start-up founders, students, and independent researchers. Stephen Merity's SHA-RNN, a compact, hybrid attention-RNN model, is designed for consumer-grade modeling as it requires significantly fewer parameters and less training time to reach near state of the art results. We analyze Merity's model here through an exploratory model analysis over several units of the architecture considering both training time and overall quality in our assessment. Ultimately, we combine these findings into a new architecture which we call SHAQ: Single Headed Attention Quasi-recurrent Neural Network. With our new architecture we achieved similar accuracy results as the SHA-RNN while accomplishing a 4x speed boost in training.

show abstract

“…Artificial intelligence and Machine learning in particular is a hot research subarea because of their numerous applications in various fields and contexts. Examples of applications include, but are not limited to: Natural Language Processing [1,2,3,4,5,6], Computer Vision [7,8,9,10], Game theory [11,12], Speech Recognition [13], Security [14,15,16,17,18,19,20,21,22,23,24], Medical diagnosis [25,26], Statistical Arbitrage [27], Network Anomaly Detection [28,29,30,31,32], Learning associations [33,34], Prediction [35,36,37,38,39], Extraction of information [40,41,42,43], Biometrics [44,45,46], Regression [47], Financial Services [48,…”

Section: Introductionmentioning

confidence: 99%

Magnetic Force Classifier: A Novel Method for Big Data Classification

et al. 2022

View full text Add to dashboard Cite

There are a plethora of invented classifiers in Machine learning literature, however, there is no optimal classifier in terms of accuracy and time taken to build the trained model, especially with the tremendous development and growth of Big data. Hence, there is still room for improvement. In this paper, we propose a new classification method that is based on the well-known magnetic force. Based on the number of points belonging to a specific class/magnet, the proposed magnetic force (MF) classifier calculates the magnetic force at each discrete point in the feature space. Unknown examples are classified using the magnetic forces recorded in the trained model by various magnets/classes. When compared to existing classifiers, the proposed MF classifier achieves comparable classification accuracy, according to the experimental results utilizing 28 different datasets. More importantly, we found that the proposed MF classifier is significantly faster than all other classifiers tested, particularly when applied to Big datasets and hence could be a viable option for structured Big data classification with some optimization.

show abstract

Simplified Self-Attention for Transformer-Based end-to-end Speech Recognition

Cited by 28 publications

References 34 publications

A Bidirectional Context Embedding Transformer for Automatic Speech Recognition

A Bidirectional Context Embedding Transformer for Automatic Speech Recognition

SHAQ: Single Headed Attention with Quasi-Recurrence

Magnetic Force Classifier: A Novel Method for Big Data Classification

Contact Info

Product

Resources

About