Dropout Regularization for Self-Supervised Learning of Transformer Encoder Speech Representation

Luo, Jian; Wang, Jianzong; Cheng, Ning; Xiao, Jing

doi:10.21437/interspeech.2021-1066

Cited by 6 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This work [13], leveraged a part of the Pan-Cancer dataset to pre-train convolutional neural networks (CNNs) for predicting survival rates in lung cancer. The researchers tackled the lack of structure in gene expression data by reformatting RNA-seq samples into gene expression images, which enabled the extraction of high-level features through CNNs.…”

Section: Related Workmentioning

confidence: 99%

Cancer Diagnosis through Contour Visualization of Gene Expression Leveraging Deep Learning Techniques

Venkatesan,

Kuppusamy Murugesan,

Chandrasekaran

et al. 2023

Diagnostics

View full text Add to dashboard Cite

Prompt diagnostics and appropriate cancer therapy necessitate the use of gene expression databases. The integration of analytical methods can enhance detection precision by capturing intricate patterns and subtle connections in the data. This study proposes a diagnostic-integrated approach combining Empirical Bayes Harmonization (EBS), Jensen–Shannon Divergence (JSD), deep learning, and contour mathematics for cancer detection using gene expression data. EBS preprocesses the gene expression data, while JSD measures the distributional differences between cancerous and non-cancerous samples, providing invaluable insights into gene expression patterns. Deep learning (DL) models are employed for automatic deep feature extraction and to discern complex patterns from the data. Contour mathematics is applied to visualize decision boundaries and regions in the high-dimensional feature space. JSD imparts significant information to the deep learning model, directing it to concentrate on pertinent features associated with cancerous samples. Contour visualization elucidates the model’s decision-making process, bolstering interpretability. The amalgamation of JSD, deep learning, and contour mathematics in gene expression dataset analysis diagnostics presents a promising pathway for precise cancer detection. This method taps into the prowess of deep learning for feature extraction while employing JSD to pinpoint distributional differences and contour mathematics for visual elucidation. The outcomes underscore its potential as a formidable instrument for cancer detection, furnishing crucial insights for timely diagnostics and tailor-made treatment strategies.

show abstract

Section: Related Workmentioning

confidence: 99%

Cancer Diagnosis through Contour Visualization of Gene Expression Leveraging Deep Learning Techniques

Venkatesan,

Kuppusamy Murugesan,

Chandrasekaran

et al. 2023

Diagnostics

View full text Add to dashboard Cite

show abstract

“…DeCoAR 2.0 [110] uses vector quantization, which is shown to improve the learned representations. Furthermore, two dropout regularization methodsattention dropout and layer dropout-are introduced with the TERA model [106], [111]. Both methods are variations on the original dropout method [112].…”

Section: B Generative Approaches 1) Motivationmentioning

confidence: 99%

Self-Supervised Speech Representation Learning: A Review

Mohamed

Lee

Borgholt

et al. 2022

IEEE J. Sel. Top. Signal Process.

169

View full text Add to dashboard Cite

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. Users may download and print one copy of any publication from the public portal for the purpose of private study or research.  You may not further distribute the material or use it for any profit-making activity or commercial gain  You may freely distribute the URL identifying the publication in the public portal If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

show abstract

“…DeCoAR 2.0 [92] uses vector quantization, which is shown to improve the learned representations. Furthermore, two dropout regularization methods-attention dropout and layer dropout-are introduced with the TERA model [93].…”

Section: U L T I -T a R G E T A P C B E S T -R Qmentioning

confidence: 99%