Direct Speech-to-Image Translation

Abstract

Direct speech-to-image translation without text is an interesting and useful topic due to the potential applications in human-computer interaction, art creation, computer-aided design. etc. Not to mention that many languages have no writing form. However, as far as we know, it has not been well-studied how to translate the speech signals into images directly and how well they can be translated. In this paper, we attempt to translate the speech signals into the image signals without the transcription stage. Specifically, a speech encoder is designed to represent the input speech signals as an embedding feature, and it is trained with a pretrained image encoder using teacher-student learning to obtain better generalization ability on new classes. Subsequently, a stacked generative adversarial network is used to synthesize high-quality images conditioned on the embedding feature. Experimental results on both synthesized and real data show that our proposed method is effective to translate the raw speech signals into images without the middle text representation. Ablation study gives more insights about our method.

Generating Images From Spoken Descriptions

“…In order to do so, we reimplemented StackGAN-v2 and replaced the text embedding with our speech embedding. Finally, we compare our results to the recently released speechbased model by [10] and the baseline used in that study, referred to as Classifier-based, in which the speech encoder was trained with the cross-entropy loss without using visual information. Specifically, in the Classifier-based method, a classifier layer is added after the speech encoder.…”

“…However, all the natural language-to-image generation research mentioned above is based on written language, i.e., text descriptions. The task most related to our work is presented in [10], in which the authors adopted the teacher-student structure to learn speech embeddings and used StackGAN-v2 [7] as the generator to generate images from the content of the speech signal. In our work, we will compare our method with this recently proposed method.…”

“…In this database, each image has one corresponding spoken caption. The work by [10], which is most closely related to our work, uses the Places database. In order to make a direct comparison with [10] on the task of S2IG, we use the same subset of the Places database as used in [10].…”

“…The work by [10], which is most closely related to our work, uses the Places database. In order to make a direct comparison with [10] on the task of S2IG, we use the same subset of the Places database as used in [10]. Specifically, this subset (referred to as the Places-subset hereafter) consists of 7 scene categories: bedroom, dinette, dining room, home office, hotel room, kitchenette, and living room, with a total of 13,803 image-spoken caption pairs for training and 2,870 image-spoken caption pairs for testing.…”

“…1 illustrates this new task. This task is similar to the independently and simultaneously proposed task of speech-to-image translation task [10].…”