Phonetic-and-Semantic Embedding of Spoken words with Applications in Spoken Content Retrieval

Abstract

Word embedding or Word2Vec has been successful in offering semantics for text words learned from the context of words. Audio Word2Vec was shown to offer phonetic structures for spoken words (signal segments for words) learned from signals within spoken words. This paper proposes a two-stage framework to perform phonetic-and-semantic embedding on spoken words considering the context of the spoken words. Stage 1 performs phonetic embedding with speaker characteristics disentangled. Stage 2 then performs semantic embedding in addition. We further propose to evaluate the phonetic-and-semantic nature of the audio embeddings obtained in Stage 2 by parallelizing with text embeddings.In general, phonetic structure and semantics inevitably disturb each other. For example the words "brother" and "sister" are close in semantics but very different in phonetic structure, while the words "brother" and "bother" are in the other way around. But phonetic-and-semantic embedding is attractive, as shown in the initial experiments on spoken document retrieval. Not only spoken documents including the spoken query can be retrieved based on the phonetic structures, but spoken documents semantically related to the query but not including the query can also be retrieved based on the semantics.Index Termsphonetic-and-semantic embedding, spoken content retrieval arXiv:1807.08089v4 [cs.CL]

Acoustically Grounded Word Embeddings for Improved Acoustics-to-word Speech Recognition

“…Recent work has explored a number of acoustic and acoustically grounded word embedding approaches. Several approaches have been developed for learning acoustic word embedding models-functions mapping arbitrary-duration spoken word signals to fixed-dimensional vectors-so as to encode either phonetic [10,11,[20][21][22][23][24] or semantic [25] information, or both [26].…”

Truly Unsupervised Acoustic Word Embeddings Using Weak Top-down Constraints in Encoder-decoder Models

“…Recent studies have therefore explored an alignment-free methodology where an arbitrary-length speech segment is embedded in a fixed-dimensional space such that segments of the same word type have similar embeddings [17][18][19][20][21][22][23][24][25]. Segments can then be compared by simply calculating a distance in this acoustic word embedding space.…”

Multilingual Acoustic Word Embedding Models for Processing Zero-resource Languages

“…Several supervised and unsupervised acoustic embedding methods have been proposed. Supervised methods include convolutional [11][12][13] and recurrent neural network (RNN) models [14][15][16][17], trained with discriminative classification and contrastive losses. Unsupervised methods include using distances to a fixed reference set [10] and unsupervised autoencoding RNNs [18][19][20].…”