Leveraging hierarchy in multimodal generative models for effective cross-modality inference

Vasco, Miguel; Yin, Hang; Melo, Francisco S.; Paiva, Ana

doi:10.1016/j.neunet.2021.11.019

Cited by 8 publications

(10 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• An unsupervised learning problem, where we learn multimodal representations on the Multimodal Handwritten Digits (MHD) dataset (Vasco et al, 2022). We showcase the geometric alignment of representations and demonstrate the superior performance of GMC compared to the baselines on a downstream classification task with missing modalities (Section 5.1);…”

Section: Methodsmentioning

confidence: 98%

“…Recently, hierarchical multimodal VAEs have been proposed to facilitate the learning of aligned multimodal representations such as Nexus (Vasco et al, 2022) and Multimodal Sensing (MUSE) (Vasco et al, 2021). Nexus considers a two-level hierarchy of modality-specific and multimodal representation spaces employing a dropout-based training scheme.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Geometric Multimodal Contrastive Representation Learning

Poklukar¹,

Vasco²,

Yin³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Learning representations of multimodal data that are both informative and robust to missing modalities at test time remains a challenging problem due to the inherent heterogeneity of data obtained from different channels. To address it, we present a novel Geometric Multimodal Contrastive (GMC) representation learning method comprised of two main components: i) a twolevel architecture consisting of modality-specific base encoder, allowing to process an arbitrary number of modalities to an intermediate representation of fixed dimensionality, and a shared projection head, mapping the intermediate representations to a latent representation space; ii) a multimodal contrastive loss function that encourages the geometric alignment of the learned representations. We experimentally demonstrate that GMC representations are semantically rich and achieve state-of-the-art performance with missing modality information on three different learning problems including prediction and reinforcement learning tasks.

show abstract

Section: Methodsmentioning

confidence: 98%

Section: Related Workmentioning

confidence: 99%

Geometric Multimodal Contrastive Representation Learning

Poklukar¹,

Vasco²,

Yin³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We compared our MLD method to MVAE [21], MMVAE [22], MOPOE [23], Hierarchical Genertive Model (NEXUS) [26], Multi-view Total Correlation Autoencoder (MVTCAE) [27], and MMVAE+ [29], re-implementing all competitors in the same code base as our method and selecting their best hyperparameters as indicated by the authors (see Appendix D for more details). For a fair comparison, we used the same encoder/decoder architecture for all models.…”

Section: Methodsmentioning

confidence: 99%

“…The Multimodal Handwritten Digits dataset (MHD) [26] contains gray-scale images of digits, the motion trajectory of the handwriting, and the sounds of the spoken digits.…”

Section: Mhdmentioning

confidence: 99%

“…However, recent comparative studies [24] have shown that none of the existing approaches fulfill the criteria of both generative quality and coherence. A variety of techniques are aimed at finding a better operating point, such as contrastive learning techniques [25], hierarchical schemes [26], total correlation-based calibration of single-modality encoders [27], and different training objectives [28]. More recently, in [29], explicitly separated shared and private latent spaces were considered as a way to overcome the aforementioned limitations.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Modal Latent Diffusion

Bounoua,

Franzese,

Michiardi

2024

Entropy

View full text Add to dashboard Cite

Multimodal datasets are ubiquitous in modern applications, and multimodal Variational Autoencoders are a popular family of models that aim to learn a joint representation of different modalities. However, existing approaches suffer from a coherence–quality tradeoff in which models with good generation quality lack generative coherence across modalities and vice versa. In this paper, we discuss the limitations underlying the unsatisfactory performance of existing methods in order to motivate the need for a different approach. We propose a novel method that uses a set of independently trained and unimodal deterministic autoencoders. Individual latent variables are concatenated into a common latent space, which is then fed to a masked diffusion model to enable generative modeling. We introduce a new multi-time training method to learn the conditional score network for multimodal diffusion. Our methodology substantially outperforms competitors in both generation quality and coherence, as shown through an extensive experimental campaign.

show abstract

Adapt and explore: Multimodal mixup for representation learning

Lin,

2024

Information Fusion

View full text Add to dashboard Cite

Leveraging hierarchy in multimodal generative models for effective cross-modality inference

Cited by 8 publications

References 22 publications

Geometric Multimodal Contrastive Representation Learning

Geometric Multimodal Contrastive Representation Learning

Multi-Modal Latent Diffusion

Adapt and explore: Multimodal mixup for representation learning

Contact Info

Product

Resources

About