MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

Ruan, Ludan; Ma, Yuanhui; Yang, Huan; Huiguo, He,; Liu, Bei; Fu, Jianlong; Yuan, Nicholas Jing; Jin, Qiong‐Hua; Guo, Baining

doi:10.48550/arxiv.2212.09478

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our model outperforms alternatives in all possible scenarios, even those that are notoriously difficult because the modalities might be only loosely correlated. We note that recent works have explored the joint generation of multiple modalities [33,34]; however, such approaches are applicationspecific, e.g., text-to-image, and essentially only target two modalities. When relevant, we compare our method to additional recent alternatives to multimodal diffusion [35,36] and show the superior performance of MLD.…”

Section: Introductionmentioning

confidence: 99%

Multi-Modal Latent Diffusion

Bounoua,

Franzese,

Michiardi

2024

Entropy

View full text Add to dashboard Cite

Multimodal datasets are ubiquitous in modern applications, and multimodal Variational Autoencoders are a popular family of models that aim to learn a joint representation of different modalities. However, existing approaches suffer from a coherence–quality tradeoff in which models with good generation quality lack generative coherence across modalities and vice versa. In this paper, we discuss the limitations underlying the unsatisfactory performance of existing methods in order to motivate the need for a different approach. We propose a novel method that uses a set of independently trained and unimodal deterministic autoencoders. Individual latent variables are concatenated into a common latent space, which is then fed to a masked diffusion model to enable generative modeling. We introduce a new multi-time training method to learn the conditional score network for multimodal diffusion. Our methodology substantially outperforms competitors in both generation quality and coherence, as shown through an extensive experimental campaign.

show abstract

Section: Introductionmentioning

confidence: 99%