NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

Wu, Chengdong; Liang, Jian; Ji, Lei; Yang, Fan; Fang, Yuejian; Jiang, Daxin; Duan, Nan

doi:10.1007/978-3-031-19787-1_41

Cited by 109 publications

(41 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Precursor to text-conditioned audio synthesis are the textconditioned image generation models, which made significant progress in quality due to architectural improvements and the availability of massive, high-quality paired training data. (Wu et al, 2022a;Hong et al, 2022;Villegas et al, 2022;Ho et al, 2022).…”

Section: Text-conditioned Image Generationmentioning

confidence: 99%

MusicLM: Generating Music From Text

Agostinelli¹,

I.²,

Borsos³

et al. 2023

Preprint

View full text Add to dashboard Cite

We introduce MusicLM, a model for generating high-fidelity music from text descriptions such as "a calming violin melody backed by a distorted guitar riff". MusicLM casts the process of conditional music generation as a hierarchical sequenceto-sequence modeling task, and it generates music at 24 kHz that remains consistent over several minutes. Our experiments show that MusicLM outperforms previous systems both in audio quality and adherence to the text descriptions. Moreover, we demonstrate that MusicLM can be conditioned on both text and a melody in that it can transform whistled and hummed melodies according to the style described in a text caption. To support future research, we publicly release MusicCaps, a dataset composed of 5.5k music-text pairs, with rich text descriptions provided by human experts.

show abstract

Section: Text-conditioned Image Generationmentioning

confidence: 99%

MusicLM: Generating Music From Text

Agostinelli¹,

I.²,

Borsos³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…According to the two timescale update rule (TTUR) [12], the learning rate is set to 0.0001 for the generator and 0.0004 for the discriminator. Following the previous text-to-image works [42,47,48,57], we adopt the Fréchet Inception Distance (FID) [12] and CLIPSIM [47] to evaluate the image fidelity and text-image semantic consistency. All GALIP models are trained on 8×3090 GPUs.…”

Section: Methodsmentioning

confidence: 99%

GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis

Tang¹,

Tang²,

Xu³

2023

Preprint

View full text Add to dashboard Cite

Synthesizing high-fidelity complex images from text is challenging. Based on large pretraining, the autoregressive and diffusion models can synthesize photo-realistic images. Although these large models have shown notable progress, there remain three flaws. 1) These models require tremendous training data and parameters to achieve good performance. 2) The multi-step generation design slows the image synthesis process heavily. 3) The synthesized visual features are difficult to control and require delicately designed prompts. To enable high-quality, efficient, fast, and controllable text-to-image synthesis, we propose Generative Adversarial CLIPs, namely GALIP. GALIP leverages the powerful pretrained CLIP model both in the discriminator and generator. Specifically, we propose a CLIP-based discriminator. The complex scene understanding ability of CLIP enables the discriminator to accurately assess the image quality. Furthermore, we propose a CLIPempowered generator that induces the visual concepts from CLIP through bridge features and prompts. The CLIPintegrated generator and discriminator boost training efficiency, and as a result, our model only requires about 3% training data and 6% learnable parameters, achieving comparable results to large pretrained autoregressive and diffusion models. Moreover, our model achieves ∼120×faster synthesis speed and inherits the smooth latent space from GAN. The extensive experimental results demonstrate the excellent performance of our GALIP. Code is available at https://github.com/tobran/GALIP.

show abstract

“…Specifically, DALL-E (Ramesh et al, 2021) demonstrates that training a large-scale auto-regressive Transformer on numerous image-text pairs can result in a high-fidelity generative model with controllable synthesis results through text prompts. NUWA (Wu et al, 2022b) presents a unified multimodal pretrained model that allows to generate or manipulate visual data (i.e., images and videos) with a 3D transformer encoder-decoder framework and a 3D Nearby Attention (3DNA) mechanism. In NUWA-Inifinity (Wu et al, 2022a), the authors further propose an autoregressive over autoregressive generation method for high-resolution infinite visual synthesis, which is capable of generating Bi-Directional Image-Text Generation.…”

Section: Vq-token-based Auto-regressive Methodsmentioning

confidence: 99%

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Gan¹,

Fu²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years. We group these approaches into three categories: (i) VLP for image-text tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding; (ii) VLP for core computer vision tasks, such as (open-set) image classification, object detection, and segmentation; and (iii) VLP for video-text tasks, such as video captioning, video-text retrieval, and video question answering. For each category, we present a comprehensive review of state-of-the-art methods, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies. In addition, for each category, we discuss advanced topics being actively explored in the research community, such as big foundation models, unified modeling, in-context few-shot learning, knowledge, robustness, and computer vision in the wild, to name a few.♠ Zhe Gan and Jianfeng Gao initiated the project. Zhe Gan and Linjie Li took lead in the writing of Chapter 1. Linjie Li and Jianfeng Gao took lead in the writing of Chapter 2. Zhe Gan further took lead in the writing of Chapter 3 and 7. Chunyuan Li took lead in the writing of Chapter 4. Linjie Li further took lead in the writing of Chapter 5. Lijuan Wang and Zicheng Liu took lead in the writing of Chapter 6. All the authors provided project advice, and contributed to paper editing and proofreading.

show abstract

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

Cited by 109 publications

References 18 publications

MusicLM: Generating Music From Text

MusicLM: Generating Music From Text

GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Contact Info

Product

Resources

About