Few-Shot Keyword Spotting With Prototypical Networks

Parnami, Archit; Lee, Minwoo

doi:10.1145/3529399.3529443

Cited by 16 publications

(9 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some works [15,20] approach the problem of KWS as a detection task. [15] designs a model architecture with multi-head attention layers and introduces soft-triple loss, which is a combination of triplet loss and softmax loss for learning feature representations.…”

Section: Related Workmentioning

confidence: 99%

“…[15] designs a model architecture with multi-head attention layers and introduces soft-triple loss, which is a combination of triplet loss and softmax loss for learning feature representations. [20] proposes metric learning-based prototypical network that can effectively extract distinctive features to detect user-defined keywords. The method still requires an additional incremental training process to adapt the model to the target user-defined keywords.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Metric Learning for User-defined Keyword Spotting

Jung¹,

Youkyum²,

Park³

et al. 2022

Preprint

View full text Add to dashboard Cite

The goal of this work is to detect new spoken terms defined by users. While most previous works address Keyword Spotting (KWS) as a closed-set classification problem, this limits their transferability to unseen terms. The ability to define custom keywords has advantages in terms of user experience.In this paper, we propose a metric learning-based training strategy for user-defined keyword spotting. In particular, we make the following contributions: (1) we construct a large-scale keyword dataset with an existing speech corpus and propose a filtering method to remove data that degrade model training;(2) we propose a metric learning-based two-stage training strategy, and demonstrate that the proposed method improves the performance on the user-defined keyword spotting task by enriching their representations; (3) to facilitate the fair comparison in the user-defined KWS field, we propose unified evaluation protocol and metrics.Our proposed system does not require an incremental training on the user-defined keywords, and outperforms previous works by a significant margin on the Google Speech Commands dataset using the proposed as well as the existing metrics.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Metric Learning for User-defined Keyword Spotting

Jung¹,

Youkyum²,

Park³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Therefore, reducing the impact of noise on performance is the focus of many speech-related tasks, including KWS. For the most typical speech task, Audio Speech Recognition (ASR), audio-visual fusion has been proven to be a promising technique to tackle the noise problem, since the visual information is not affected by acoustic distortions [13][14][15][16][17][18][19][20][21]. As the most common forms of perception in human communication, hearing and watching have received increasing attention from researchers in the multi-modal field.…”

Section: Introductionmentioning

confidence: 99%

Audio–visual keyword transformer for unconstrained sentence‐level keyword spotting

Jia-le

Wang

et al. 2023

CAAI Trans on Intel Tech

View full text Add to dashboard Cite

As one of the most effective methods to improve the accuracy and robustness of speech tasks, the audio–visual fusion approach has recently been introduced into the field of Keyword Spotting (KWS). However, existing audio–visual keyword spotting models are limited to detecting isolated words, while keyword spotting for unconstrained speech is still a challenging problem. To this end, an Audio–Visual Keyword Transformer (AVKT) network is proposed to spot keywords in unconstrained video clips. The authors present a transformer classifier with learnable CLS tokens to extract distinctive keyword features from the variable‐length audio and visual inputs. The outputs of audio and visual branches are combined in a decision fusion module. As humans can easily notice whether a keyword appears in a sentence or not, our AVKT network can detect whether a video clip with a spoken sentence contains a pre‐specified keyword. Moreover, the position of the keyword is localised in the attention map without additional position labels. Experimental results on the LRS2‐KWS dataset and our newly collected PKU‐KWS dataset show that the accuracy of AVKT exceeded 99% in clean scenes and 85% in extremely noisy conditions. The code is available at https://github.com/jialeren/AVKT.

show abstract

“…Although existing predefined KWS models show high detection performance [1,2,3], the necessity of a large dataset containing target keywords and inflexibility of changing target keywords hinder KWS models from expanding to various applications. When it comes to userdefined KWS, users can customize the target keywords with only a few enrollment samples [4,5,6,7] or in the form of string [8,9]. Few-shot KWS (FS-KWS) especially has shown its feasibility through meta learning [4], transfer learning [5], and metric learning [6,7], operating on the few-shot detection scenario.…”

Section: Introductionmentioning

confidence: 99%

“…When it comes to userdefined KWS, users can customize the target keywords with only a few enrollment samples [4,5,6,7] or in the form of string [8,9]. Few-shot KWS (FS-KWS) especially has shown its feasibility through meta learning [4], transfer learning [5], and metric learning [6,7], operating on the few-shot detection scenario. These approaches typically require learning from a large corpus with lots of different keywords to secure generalization on unseen keywords with few samples.…”

Section: Introductionmentioning

confidence: 99%

Fully Unsupervised Training of Few-shot Keyword Spotting

Lee¹,

Kim²,

Mun³

et al. 2022

Preprint

View full text Add to dashboard Cite

For training a few-shot keyword spotting (FS-KWS) model, a large labeled dataset containing massive target keywords has known to be essential to generalize to arbitrary target keywords with only a few enrollment samples. To alleviate the expensive data collection with labeling, in this paper, we propose a novel FS-KWS system trained only on synthetic data. The proposed system is based on metric learning enabling target keywords to be detected using distance metrics. Exploiting the speech synthesis model that generates speech with pseudo phonemes instead of texts, we easily obtain a large collection of multi-view samples with the same semantics. These samples are sufficient for training, considering metric learning does not intrinsically necessitate labeled data. All of the components in our framework do not require any supervision, making our method unsupervised. Experimental results on real datasets show our proposed method is competitive even without any labeled and real datasets.

show abstract

Few-Shot Keyword Spotting With Prototypical Networks

Cited by 16 publications

References 7 publications

Metric Learning for User-defined Keyword Spotting

Metric Learning for User-defined Keyword Spotting

Audio–visual keyword transformer for unconstrained sentence‐level keyword spotting

Fully Unsupervised Training of Few-shot Keyword Spotting

Contact Info

Product

Resources

About