Speech Technology for Unwritten Languages

Scharenborg, Odette; Ondel, Lucas; Palaskar, Shruti; Arthur, Philip; Ciannella, Francesco; Du, Mingxing; Larsen, Elin; Merkx, Danny; Riad, Rachid; Wang, Liming; Dupoux, Emmanuel; Besacier, Laurent; Black, Alan W.; Hasegawa‐Johnson, Mark; Metze, Florian; Neubig, Graham; Stüker, Sebastian; Godard, Pierre; Müller, Markus

doi:10.1109/taslp.2020.2973896

Cited by 21 publications

(20 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Typically, a well-trained deep neural network (DNN) based ASR system [2] requires hundreds to thousands of hours of transcribed speech data. This requirement can be challenging to most of the world's spoken languages that are considered low-resourced [3]. Most languages, including ethnic minority languages in most countries outside Europe, lack transcribed speech data, digitized texts, and digitized pronunciation lexicons.…”

Section: Introductionmentioning

confidence: 99%

How Phonotactics Affect Multilingual and Zero-Shot ASR Performance

Feng

Żelasko

Moro-Velázquez

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

The idea of combining multiple languages' recordings to train a single automatic speech recognition (ASR) model brings the promise of the emergence of universal speech representation. Recently, a Transformer encoder-decoder model has been shown to leverage multilingual data well in IPA transcriptions of languages presented during training. However, the representations it learned were not successful in zero-shot transfer to unseen languages. Because that model lacks an explicit factorization of the acoustic model (AM) and language model (LM), it is unclear to what degree the performance suffered from differences in pronunciation or the mismatch in phonotactics. To gain more insight into the factors limiting zero-shot ASR transfer, we replace the encoder-decoder with a hybrid ASR system consisting of a separate AM and LM. Then, we perform an extensive evaluation of monolingual, multilingual, and crosslingual (zeroshot) acoustic and language models on a set of 13 phonetically diverse languages. We show that the gain from modeling crosslingual phonotactics is limited, and imposing a too strong model can hurt the zero-shot transfer. Furthermore, we find that a multilingual LM hurts a multilingual ASR system's performance, and retaining only the target language's phonotactic data in LM training is preferable.

show abstract

Section: Introductionmentioning

confidence: 99%

How Phonotactics Affect Multilingual and Zero-Shot ASR Performance

Feng

Żelasko

Moro-Velázquez

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recent research on VGS models focused on architectural and training scheme improvements [5], [6], [11]and applications such as semantic keyword spotting [7], [12] and speech-based image retrieval [3], [5], [6], [8]. Recent research has shown that VGS models implicitly learn to recognise meaningful sentence constituents such as phonemes and words and the presence of these constituents can be decoded from the speech embeddings [5], [6], [13]- [15].…”

Section: Introductionmentioning

confidence: 99%

Learning to Recognise Words Using Visually Grounded Speech

Scholten

Merkx

Scharenborg

2021

2021 IEEE International Symposium on Circuits and Systems (ISCAS)

Self Cite

View full text Add to dashboard Cite

We investigated word recognition in a Visually Grounded Speech model. The model has been trained on pairs of images and spoken captions to create visually grounded embeddings which can be used for speech to image retrieval and vice versa. We investigate whether such a model can be used to recognise words by embedding isolated words and using them to retrieve images of their visual referents. We investigate the timecourse of word recognition using a gating paradigm and perform a statistical analysis to see whether well known word competition effects in human speech processing influence word recognition. Our experiments show that the model is able to recognise words, and the gating paradigm reveals that words can be recognised from partial input as well and that recognition is negatively influenced by word competition from the word initial cohort.

show abstract

“…Instead, the supervisory information consists of the corresponding images of the speech descriptions. Inspired by human infants' ability to learn spoken language by listening and paying attention to the concurrent speech and visual scenes, several recent methods [15], [36], [37], [38], [39], [40] have been proposed to learn speech models grounded by visual information.…”

Section: B Visually-grounded Speech Embedding Learningmentioning

confidence: 99%

“…With speech as input, the trained speech encoder is used to extract the speech embeddings. Different from most of the previously described visually-grounded speech embedding learning methods in which the traditional triplet loss function was adopted to train the model [15], [36], [37], [39], [40], in this work we propose a more effective matching loss which is similar to the masked margin softmax loss [38] that can make full use of negative samples within a minibatch. Furthermore, previous studies only focused on databases with scene images, primarily Flickr8k [16], while no finegrained image databases, e.g., CUB-200 [12], in which images from different categories share similar semantics, have been considered.…”

Section: B Visually-grounded Speech Embedding Learningmentioning

confidence: 99%

Generating Images From Spoken Descriptions

Wang

Qiao

Zhu

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

Text-based technologies, such as text translation from one language to another, and image captioning, are gaining popularity. However, approximately half of the world's languages are estimated to be lacking a commonly used written form. Consequently, these languages cannot benefit from text-based technologies. This paper presents 1) a new speech technology task, i.e., a speech-to-image generation (S2IG) framework which translates speech descriptions to photo-realistic images 2) without using any text information, thus allowing unwritten languages to potentially benefit from this technology. The proposed speechto-image framework, referred to as S2IGAN, consists of a speech embedding network and a relation-supervised denselystacked generative model. The speech embedding network learns speech embeddings with the supervision of corresponding visual information from images. The relation-supervised denselystacked generative model synthesizes images, conditioned on the speech embeddings produced by the speech embedding network, that are semantically consistent with the corresponding spoken descriptions. Extensive experiments are conducted on four public benchmark databases: two databases that are commonly used in text-to-image generation tasks, i.e., CUB-200 and Oxford-102 for which we created synthesized speech descriptions, and two databases with natural speech descriptions which are often used in the field of cross-modal learning of speech and images, i.e., Flickr8k and Places. Results on these databases demonstrate the effectiveness of the proposed S2IGAN on synthesizing highquality and semantically-consistent images from the speech signal, yielding a good performance and a solid baseline for the S2IG task.

show abstract

Speech Technology for Unwritten Languages

Cited by 21 publications

References 38 publications

How Phonotactics Affect Multilingual and Zero-Shot ASR Performance

How Phonotactics Affect Multilingual and Zero-Shot ASR Performance

Learning to Recognise Words Using Visually Grounded Speech

Generating Images From Spoken Descriptions

Contact Info

Product

Resources

About