Mutual-learning sequence-level knowledge distillation for automatic speech recognition

Li, Zerui; Ming, Yue; Lei, Yang; Xue, Jing-Hao

doi:10.1016/j.neucom.2020.11.025

Cited by 20 publications

(4 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Knowledge distillation has proven effective in various domains, such as natural language processing[45, 46, 47, 48], computer vision[41, 49, 50, 51], and speech recognition[52, 53, 54, 55]. Its versatility stems from its capacity to distill the rich knowledge captured by a complex model into a more compact representation, which is suitable for deployment in environments with limited resources.…”

Section: Discussionmentioning

confidence: 99%

epiTCR-KDA: Knowledge Distillation model on Dihedral Angles for TCR-peptide prediction

Pham,

Su,

Nguyen

et al. 2024

Preprint

View full text Add to dashboard Cite

Motivation: Antigen recognition by T-cell receptors (TCRs) triggers cascades of immune responses. Successful predictions of the TCR and antigen (as peptide) bindings therefore signify the advancements in immunotherapy. However, most of current TCR-peptide interaction predictors fail to predict unseen data. This limitation may be derived from the conventional usage of TCR and/or peptide sequences as input, which may not adequately reflect their structural characteristics. Therefore, incorporating the TCR and peptide structural information into the prediction model to improve the generalizability is necessary. Results: We presented epiTCR-KDA as a new predictor of TCR-peptide binding that utilises structural information, specifically the dihedral angles between the residues of both the peptide and the TCR. This structural descriptor was integrated into a model constructed using knowledge distillation to enhance its generalizability. The epiTCR-KDA demonstrated competitive prediction performance, with an AUC of 0.99 for seen data and AUC of 0.86 for unseen data. Across multiple public datasets, epiTCR-KDA consistently outperformed other predictors, such as epiTCR, NetTCR, BERTrand, TEIM-Seq, TEINet, and ImRex, maintaining a median AUC of 0.9 (ranging from 0.82 to 0.91). Further analysis of epiTCR-KDA performance indicated that the cosine similarity of the dihedral angle vectors between the unseen testing data and training data is crucial for its stable performance. In conclusion, our epiTCR-KDA model, with its capacity to predict for unseen data, has brought us one step closer toward the development of a highly effective pipeline for affordable antigen-based immunotherapy. Availability and implementation: epiTCR-KDA is available on GitHub (https://github.com/ddiem-ri-4D/epiTCR-KDA) .

show abstract

Section: Discussionmentioning

confidence: 99%

epiTCR-KDA: Knowledge Distillation model on Dihedral Angles for TCR-peptide prediction

Pham,

Su,

Nguyen

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, end-to-end models were considered in deep learning for ASR. However, these algorithms still are not suitable for real-time ASR applications because of their large model sizes and computation complexity [65]. On the contrary, SNNs models and algorithms offer challenging tools for temporal tasks such as ASR because they can directly deal with temporal features besides spatial features.…”

Section: Related Workmentioning

confidence: 99%

Automated Adaptive Threshold-Based Feature Extraction and Learning for Spiking Neural Networks

Amin

2021

IEEE Access

View full text Add to dashboard Cite

Over the past years Spiking Neural Networks (SNNs) models became attractive as a possible bridge to enable low-power event-driven neuromorphic hardware. SNNs have a high computational power due to the implicit employment of the biologically inspired input times. SNNs employ various parameters such as neuron threshold, synaptic delays, and weights in their structures. However, SNNs applications are still limited and elementary compared with other neural network architectures such as the Convolution Neural Networks (CNNs). In this research, a new SNN-based model named Adaptive Threshold Module (ATM) and its algorithm are proposed. The proposed ATM and algorithm depend on the adaptation of the internal spiking neuron threshold level. Adapting the threshold of the neurons is employed to control the spiking neuron firing rate to uniquely extract the main features of the input pattern that is in the shape of spike trains. It is shown that this technique works as an automated feature extraction method of input patterns in an efficient and faster way than other methods. The proposed method can preserve all information of the input spike trains. Simulations of the proposed model and the algorithm, using the challenging speech TIDIGITS dataset, sound RWCP dataset, and Poisson distribution spike trains, show encouraging results. The ATM can make SNN provide an accuracy surpassing that of the current state-of-the-art SNN algorithms and conventional non-spiking learning models.

show abstract

“…But this method requires a large amount of corpus for training, which is time-consuming and not accurate. Finally, the human-machine interaction method of deep learning is utilized, which automatically extracts speech and text features through deep learning models, effectively improving the performance of language recognition [4]. With the increasing demand of people, the human-machine environment has become even more harsh, so simple deep learning models can no longer meet high requirements for completing human-machine interaction tasks.…”

Section: Introductionmentioning

confidence: 99%

Robot Human-Machine Interaction Method Based on Natural Language Processing and Speech Recognition

Wang,

Long

2023

IJACSA

View full text Add to dashboard Cite

With the rapid development of artificial intelligence technology, robots have gradually entered people's lives and work. The robot human-machine interaction system for image recognition has been widely used. However, there are still many problems with robot human-machine interaction methods that utilize natural language processing and speech recognition. Therefore, this study proposes a new robot human-machine interaction method that combines structured perceptron lexical analysis model and transfer dependency syntactic analysis model on the basis of existing interaction systems. The purpose is to further explore language based human-machine interaction systems and improve interaction performance. The experiment shows that the testing accuracy of the structured perceptron model reaches 95%, the recall rate reaches 81%, and the F1 value reaches 82%. The transfer dependency syntax analysis model has a data analysis speed of up to 750K/s. In simulation testing, the new robot human-machine interaction method has an accuracy of 92% compared to other existing methods, and exhibits excellent robustness and response sensitivity. In summary, research methods can provide a theoretical and practical basis for the improvement of robot interaction capabilities and the further development of human-machine collaboration.

show abstract

Mutual-learning sequence-level knowledge distillation for automatic speech recognition

Cited by 20 publications

References 46 publications

epiTCR-KDA: Knowledge Distillation model on Dihedral Angles for TCR-peptide prediction

epiTCR-KDA: Knowledge Distillation model on Dihedral Angles for TCR-peptide prediction

Automated Adaptive Threshold-Based Feature Extraction and Learning for Spiking Neural Networks

Robot Human-Machine Interaction Method Based on Natural Language Processing and Speech Recognition

Contact Info

Product

Resources

About