Sound-Word2Vec: Learning Word Representations Grounded in Sounds

Vijayakumar, Ashwin K.; Vedantam, Ramakrishna; Parikh, Devi

doi:10.18653/v1/d17-1096

Cited by 19 publications

(13 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In natural language processing, images have also been used to capture aspects of meaning (semantics) of written language; see [31,32] for reviews. Other studies have considered multimodal modelling of sounds (not speech) with text and images [33][34][35], and phonemes with images [36]. classifier (disregarding the order and quantity of words).…”

Section: Related Workmentioning

confidence: 99%

Visually Grounded Learning of Keyword Prediction from Untranscribed Speech

Kamper

Settle²,

Shakhnarovich

et al. 2017

Interspeech 2017

View full text Add to dashboard Cite

During language acquisition, infants have the benefit of visual cues to ground spoken language. Robots similarly have access to audio and visual sensors. Recent work has shown that images and spoken captions can be mapped into a meaningful common space, allowing images to be retrieved using speech and vice versa. In this setting of images paired with untranscribed spoken captions, we consider whether computer vision systems can be used to obtain textual labels for the speech. Concretely, we use an image-to-words multi-label visual classifier to tag images with soft textual labels, and then train a neural network to map from the speech to these soft targets. We show that the resulting speech system is able to predict which words occur in an utterance-acting as a spoken bag-of-words classifier-without seeing any parallel speech and text. We find that the model often confuses semantically related words, e.g. "man" and "person", making it even more effective as a semantic keyword spotter.

show abstract

Section: Related Workmentioning

confidence: 99%

Visually Grounded Learning of Keyword Prediction from Untranscribed Speech

Kamper

Settle²,

Shakhnarovich

et al. 2017

Interspeech 2017

View full text Add to dashboard Cite

show abstract

“…In their work, a 'bag-of-audio-words' approach is used, in which auditory grounding is achieved by dividing sound files into frames, clustering these frames as ''audio words'' and subsequently quantizing them into representations by comparing frame descriptors with the centroids. Recently, Vijayakumar et al [30] proposed an embedding scheme that learns specialized word embeddings grounded in sounds by using a variety of audio features. These techniques were found to work well for modeling human similarity and relatedness judgments and related experiments.…”

Section: Auditory Representationsmentioning

confidence: 99%

A Synchronized Word Representation Method With Dual Perceptual Information

Zhu

et al. 2020

IEEE Access

View full text Add to dashboard Cite

The information used for human natural language comprehension is usually perceptual information, such as text, sounds, and images. In recent years, language models that learn semantics from single perceptual information sources (text) have gradually developed into multimodal language models that learn semantics from multiple perceptual information sources. Sound is perceptual information other than text that has been proven effective by many related works. However, there is still a need for further research on the incorporation method for perceptual information. Thus, this paper proposes a language model that synchronously trains dual perceptual information to enhance word representation. The representation is trained in a synchronized way that adopts an attention model to utilize both text and phonetic perceptual information in unsupervised learning tasks. On basis of that, these dual perceptual information is processed simultaneously, and that is similar with the cognitive process of human language understanding. The experiment results show that our approach achieve superior results in text classification and word similarity tasks with four languages of data set. INDEX TERMS Information representation, multi-layer neural network, natural language processing, unsupervised learning.

show abstract

“…Auditory grounding was achieved by dividing sound files into frames, clustering these as "audio words" and subsequently quantizing them into representations by comparing frame descriptors with the centroids. More recently, Vijayakumar, Vedantam, and Parikh (2017) proposed an embedding scheme that learns specialized word embeddings grounded in sounds, using a variety of audio features. These techniques were found to work well for modeling human similarity and relatedness judgments and related experiments.…”

Section: Auditory Representationsmentioning

confidence: 99%

Learning Neural Audio Embeddings for Grounding Semantics in Auditory Perception

Kiela¹,

Clark²

2017

jair

View full text Add to dashboard Cite

Multi-modal semantics, which aims to ground semantic representations in perception, has relied on feature norms or raw image data for perceptual input. In this paper we examine grounding semantic representations in raw auditory data, using standard evaluations for multi-modal semantics. After having shown the quality of such auditorily grounded representations, we show how they can be applied to tasks where auditory perception is relevant, including two unsupervised categorization experiments, and provide further analysis. We find that features transfered from deep neural networks outperform bag of audio words approaches. To our knowledge, this is the first work to construct multi-modal models from a combination of textual information and auditory information extracted from deep neural networks, and the first work to evaluate the performance of tri-modal (textual, visual and auditory) semantic models.

show abstract

Sound-Word2Vec: Learning Word Representations Grounded in Sounds

Cited by 19 publications

References 19 publications

Visually Grounded Learning of Keyword Prediction from Untranscribed Speech

Visually Grounded Learning of Keyword Prediction from Untranscribed Speech

A Synchronized Word Representation Method With Dual Perceptual Information

Learning Neural Audio Embeddings for Grounding Semantics in Auditory Perception

Contact Info

Product

Resources

About