Large-Scale Representation Learning from Visually Grounded Untranscribed Speech

Ilharco, Gabriel; Zhang, Yuan; Baldridge, Jason

doi:10.18653/v1/k19-1006

Cited by 58 publications

(62 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As text pretraining schemes seem to be reaching the point of diminishing returns, even for some syntactic phenomena (van Schijndel et al, 2019), we posit that other forms of supervision, such as multimodal perception (Ilharco et al, 2019), are necessary to learn the remaining aspects of meaning in context. Learning by observation should not be a purely linguistic process, since leveraging and combining the patterns of multimodal perception can combinatorially boost the amount of signal in data through cross-referencing and synthesis.…”

Section: Ws2: the Written Worldmentioning

confidence: 99%

Experience Grounds Language

Bisk¹,

Holtzman²,

Thomason³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

164

136

View full text Add to dashboard Cite

Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates. Despite the incredible effectiveness of language processing models to tackle tasks after being trained on text alone, successful linguistic communication relies on a shared experience of the world. It is this shared experience that makes utterances meaningful.Natural language processing is a diverse field, and progress throughout its development has come from new representational theories, modeling techniques, data collection paradigms, and tasks. We posit that the present success of representation learning approaches trained on large, text-only corpora requires the parallel tradition of research on the broader physical and social context of language to address the deeper questions of communication.

show abstract

Section: Ws2: the Written Worldmentioning

confidence: 99%

Experience Grounds Language

Bisk¹,

Holtzman²,

Thomason³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

164

136

View full text Add to dashboard Cite

show abstract

“…Possible directions for future work include: (1) OCR postprocessing improvement: spelling correction, noise removal, etc. ; (2) enhancing text-based prediction and/or object detection using thesauri, knowledge graphs, e.g., replacing specific entities with hypernyms similar to Ilharco et al (2019) or word associations datasets, e.g., Wordgame (Louwe, 2020); (3) developing a joint architecture for images and texts: obviously, simple blending might not be the best choice for the task.…”

Section: Discussionmentioning

confidence: 99%

Ad Lingua: Text Classification Improves Symbolism Prediction in Image Advertisements

Savchenko¹,

Alekseev²,

Kwon³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Understanding image advertisements is a challenging task, often requiring non-literal interpretation. We argue that standard image-based predictions are insufficient for symbolism prediction. Following the intuition that texts and images are complementary in advertising, we introduce a multimodal ensemble of a state of the art image-based classifier, a classifier based on an object detection architecture, and a fine-tuned language model applied to texts extracted from ads by OCR. The resulting system establishes a new state of the art in symbolism prediction.

show abstract

“…Instead, the supervisory information consists of the corresponding images of the speech descriptions. Inspired by human infants' ability to learn spoken language by listening and paying attention to the concurrent speech and visual scenes, several recent methods [15], [36], [37], [38], [39], [40] have been proposed to learn speech models grounded by visual information.…”

Section: B Visually-grounded Speech Embedding Learningmentioning

confidence: 99%

Generating Images From Spoken Descriptions

Wang

Qiao

Zhu

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Text-based technologies, such as text translation from one language to another, and image captioning, are gaining popularity. However, approximately half of the world's languages are estimated to be lacking a commonly used written form. Consequently, these languages cannot benefit from text-based technologies. This paper presents 1) a new speech technology task, i.e., a speech-to-image generation (S2IG) framework which translates speech descriptions to photo-realistic images 2) without using any text information, thus allowing unwritten languages to potentially benefit from this technology. The proposed speechto-image framework, referred to as S2IGAN, consists of a speech embedding network and a relation-supervised denselystacked generative model. The speech embedding network learns speech embeddings with the supervision of corresponding visual information from images. The relation-supervised denselystacked generative model synthesizes images, conditioned on the speech embeddings produced by the speech embedding network, that are semantically consistent with the corresponding spoken descriptions. Extensive experiments are conducted on four public benchmark databases: two databases that are commonly used in text-to-image generation tasks, i.e., CUB-200 and Oxford-102 for which we created synthesized speech descriptions, and two databases with natural speech descriptions which are often used in the field of cross-modal learning of speech and images, i.e., Flickr8k and Places. Results on these databases demonstrate the effectiveness of the proposed S2IGAN on synthesizing highquality and semantically-consistent images from the speech signal, yielding a good performance and a solid baseline for the S2IG task.

show abstract

Large-Scale Representation Learning from Visually Grounded Untranscribed Speech

Cited by 58 publications

References 56 publications

Experience Grounds Language

Experience Grounds Language

Ad Lingua: Text Classification Improves Symbolism Prediction in Image Advertisements

Generating Images From Spoken Descriptions

Contact Info

Product

Resources

About