Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network

Huang, Kaixun; Zhang, Ao; Zhanheng, Yang,; Guo, Pengcheng; Mu, Bingshen; Xu, Tianyi; Xie, Lihua

doi:10.21437/interspeech.2023-767

Cited by 9 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Keyword extraction is performed using the KeyBert [32] technique, and contextual ASR is employed to enhance recognition for specific terms. Our benchmark system utilizes the contextual phrase prediction network (CPP network) ASR [33], as the contextual ASR model.…”

Section: Benchmark Systemmentioning

confidence: 99%

See 1 more Smart Citation

Assembling Alibaba:

Lin¹

2021

Engaging Social Media in China

View full text Add to dashboard Cite

Masked Language Modeling (MLM) is widely used to pretrain language models. The standard random masking strategy in MLM causes the pre-trained language models (PLMs) to be biased towards high-frequency tokens. Representation learning of rare tokens is poor and PLMs have limited performance on downstream tasks. To alleviate this frequency bias issue, we propose two simple and effective Weighted Sampling strategies for masking tokens based on token frequency and training loss. We apply these two strategies to BERT and obtain Weighted-Sampled BERT (WSBERT). Experiments on the Semantic Textual Similarity benchmark (STS) show that WSBERT significantly improves sentence embeddings over BERT. Combining WSBERT with calibration methods and prompt learning further improves sentence embeddings. We also investigate fine-tuning WSBERT on the GLUE benchmark and show that Weighted Sampling also improves the transfer learning capability of the backbone PLM. We further analyze and provide insights into how WSBERT improves token embeddings.

show abstract

Section: Benchmark Systemmentioning

confidence: 99%

“…In the original CPP paper [33], a simulation approach randomly selects three phrases containing 1-3 words from the speech transcript. These phrases are combined with distractors and <nobias>to form the contextual biasing list for training.…”

Section: Training Bias List Generationmentioning

confidence: 99%