Multilingual Transfer of Acoustic Word Embeddings Improves When Training on Languages Related to the Target Zero-Resource Language

Jacobs, Christiaan; Kamper, Herman

doi:10.21437/interspeech.2021-461

Cited by 8 publications

(6 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Learning A recent study on zero-resource AWEs has shown that cross-lingual transfer is more successful when the source (L1) and target (L2) languages are more related (Jacobs and Kamper, 2021). We conducted a preliminary experiment on the cross-lingual word discrimination performance of the models in our study and observed a similar effect.…”

Section: Implications On Cross-lingual Transfersupporting

confidence: 58%

How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings

Abdullah¹,

Zaitova²,

Avgustinova³

et al. 2021

Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

How do neural networks "perceive" speech sounds from unknown languages? Does the typological similarity between the model's training language (L1) and an unknown language (L2) have an impact on the model representations of L2 speech signals? To answer these questions, we present a novel experimental design based on representational similarity analysis (RSA) to analyze acoustic word embeddings (AWEs)-vector representations of variable-duration spoken-word segments. First, we train monolingual AWE models on seven Indo-European languages with various degrees of typological similarity. We then employ RSA to quantify the cross-lingual similarity by simulating native and non-native spoken-word processing using AWEs. Our experiments show that typological similarity indeed affects the representational similarity of the models in our study. We further discuss the implications of our work on modeling speech processing and language similarity with neural networks.

show abstract

Section: Implications On Cross-lingual Transfersupporting

confidence: 58%

How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings

Abdullah¹,

Zaitova²,

Avgustinova³

et al. 2021

Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

show abstract

“…We conduct an intrinsic evaluation for the AWEs to assess the performance of our models using the same-different acoustic word discrimination task with the mean average precision (mAP) metric [31,38,39]. Prior work has shown that performance on this task positively correlates with improvement on downstream QbE speech search [32]. This task evaluates the ability of the model to determine whether two given speech segments correspond to the same word type-that is, whether or not two acoustic segments are exemplars of the same category.…”

Section: Resultsmentioning

confidence: 99%

“…We compare the performance of our proposed model to a strong baseline that explicitly minimizes the distance between exemplars of the same lexical category. The baseline model employs a contrastive triplet loss that has been extensively explored in the AWEs literature with different underlying architectures and has shown strong discriminative performance [9,[30][31][32]. Given a matching pair of AWEs (x a , x + )-i.e., embeddings of two exemplars of the same word type-the objective is then to minimize a triplet margin loss…”

Section: Baseline: Contrastive Acoustic Modelmentioning

confidence: 99%

Integrating Form and Meaning: A Multi-Task Learning Model for Acoustic Word Embeddings

Abdullah¹,

Möbius²,

Klakow³

2022

Preprint

View full text Add to dashboard Cite

Models of acoustic word embeddings (AWEs) learn to map variable-length spoken word segments onto fixed-dimensionality vector representations such that different acoustic exemplars of the same word are projected nearby in the embedding space. In addition to their speech technology applications, AWE models have been shown to predict human performance on a variety of auditory lexical processing tasks. Current AWE models are based on neural networks and trained in a bottom-up approach that integrates acoustic cues to build up a word representation given an acoustic or symbolic supervision signal. Therefore, these models do not leverage or capture high-level lexical knowledge during the learning process. In this paper, we propose a multi-task learning model that incorporates top-down lexical knowledge into the training procedure of AWEs. Our model learns a mapping between the acoustic input and a lexical representation that encodes high-level information such as word semantics in addition to bottom-up form-based supervision. We experiment with three languages and demonstrate that incorporating lexical knowledge improves the embedding space discriminability and encourages the model to better separate lexical categories.

show abstract

“…In addition, we also explore pooling functions with trainable parameters, such as in [6,7,10]. We follow [9,10,19] and train the pooling function g with a contrastive loss. Specifically, we use NTXent [20] which is defined as…”

Section: Task Overviewmentioning

confidence: 99%

“…Most previous work on constructing unsupervised AWEs has approached the problem using learned pooling, where positive training pairs of similar speech segments (assumed to be the same word or n-gram) are used to learn a pooling function, based on a reconstruction [6,7,8] or contrastive [9,10] objective. Despite good AWE quality, these methods rely on identifying positive training pairs from a corpus using k-nearestneighbors methods [10,11].…”

Section: Introductionmentioning

confidence: 99%

Acoustic Word Embeddings for Untranscribed Target Languages with Continued Pretraining and Learned Pooling

Sanabria¹,

Klejch²,

Tang³

et al. 2023

Interspeech 2023

View full text Add to dashboard Cite

Acoustic word embeddings are typically created by training a pooling function using pairs of word-like units. For unsupervised systems, these are mined using k-nearest neighbor (KNN) search, which is slow. Recently, mean-pooled representations from a pre-trained self-supervised English model were suggested as a promising alternative, but their performance on target languages was not fully competitive. Here, we explore improvements to both approaches: we use continued pre-training to adapt the self-supervised model to the target language, and we use a multilingual phone recognizer (MPR) to mine phone n-gram pairs for training the pooling function. Evaluating on four languages, we show that both methods outperform a recent approach on word discrimination. Moreover, the MPR method is orders of magnitude faster than KNN, and is highly data efficient. We also show a small improvement from performing learned pooling on top of the continued pre-trained representations.

show abstract

Multilingual Transfer of Acoustic Word Embeddings Improves When Training on Languages Related to the Target Zero-Resource Language

Cited by 8 publications

References 52 publications

How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings

How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings

Integrating Form and Meaning: A Multi-Task Learning Model for Acoustic Word Embeddings

Acoustic Word Embeddings for Untranscribed Target Languages with Continued Pretraining and Learned Pooling

Contact Info

Product

Resources

About