Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis

Seichter, Daniel; Köhler, Mona; Lewandowski, Benjamin; Wengefeld, Tim; Groß, Horst–Michael

doi:10.48550/arxiv.2011.06961

Cited by 7 publications

(14 citation statements)

References 37 publications

(65 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multimodal Learning Deep learning makes fusing different signals easier, which enable us to develop many multimodal frameworks. For example, [26,15,13,20,12] combine RGB and depth images to improve semantic segmentation; [7,11] fuse audio with video to do scene understanding; researchers also explore audio-visual source separation and localization [36,10]. In semi-supervised setting, [27] proposed a novel method, Total Correlation Gain Maximization (TCGM), based on information theory, which explores the information intersection by maximizing the total correlation gain among all the modalities.…”

Section: Related Workmentioning

confidence: 99%

What Makes Multi-modal Learning Better than Single (Provably)

Huang¹,

Du²,

Xue³

et al. 2021

Preprint

View full text Add to dashboard Cite

The world provides us with data of multiple modalities. Intuitively, models fusing data from different modalities outperform unimodal models, since more information is aggregated. Recently, joining the success of deep learning, there is an influential line of work on deep multimodal learning, which has remarkable empirical results on various applications. However, theoretical justifications in this field are notably lacking. Can multimodal provably perform better than unimodal? In this paper, we answer this question under a most popular multimodal learning framework, which firstly encodes features from different modalities into a common latent space and seamlessly maps the latent representations into the task space. We prove that learning with multiple modalities achieves a smaller population risk than only using its subset of modalities. The main intuition is that the former has more accurate estimate of the latent space representation. To the best of our knowledge, this is the first theoretical treatment to capture important qualitative phenomena observed in real multimodal applications. Combining with experiment results, we show that multimodal learning does possess an appealing formal guarantee.

show abstract

Section: Related Workmentioning

confidence: 99%

What Makes Multi-modal Learning Better than Single (Provably)

Huang¹,

Du²,

Xue³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Implementation details. Our segmentation UMT is based on a state-of-the-art method, i.e., ESANet [41]. For RGB modality, the ResNet34 backbone, downsampling method, and contextual module are employed following [31,49].…”

Section: Rgb-depth Semantic Segmentation Experimentsmentioning

confidence: 99%

“…Much current research on multi-modal fusion mainly revolves around the design of model architectures, such as middle fusion [41,45], late fusion [44] and attention-based fusion [14,41]. However, simply combining multiple modalities often results in unsatisfactory performance.…”

Section: Introductionmentioning

confidence: 99%

Improving Multi-Modal Learning with Uni-Modal Teachers

Du,

Li,

Liu

et al. 2021

Preprint

View full text Add to dashboard Cite

Learning multi-modal representations is an essential step towards real-world robotic applications, and various multi-modal fusion models have been developed for this purpose. However, we observe that existing models, whose objectives are mostly based on joint training, often suffer from learning inferior representations of each modality. We name this problem Modality Failure, and hypothesize that the imbalance of modalities and the implicit bias of common objectives in fusion method prevent encoders of each modality from sufficient feature learning. To this end, we propose a new multi-modal learning method, Uni-Modal Teacher, which combines the fusion objective and uni-modal distillation to tackle the modality failure problem. We show that our method not only drastically improves the representation of each modality, but also improves the overall multi-modal task performance. Our method can be effectively generalized to most multi-modal fusion approaches. We achieve more than 3% improvement on the VGGSound audio-visual classification task, as well as improving performance on the NYU depth V2 RGB-D image segmentation task.

show abstract

“…The data augmentation strategies we adopted include random cropping, rotation, and color jittering. We use ESANet [28], an efficient ResNet-based encoder, as our backbone. We use the common 40-class label setting and mean IoU(mIoU) as the evaluation metric.…”

Section: Nyuv2 Semantic Segmentationmentioning

confidence: 99%

“…In addition, we include an InfoNCE [24] baseline where we directly contrast multimodal input tuples without tuple disturbing and sample optimization. We also include supervised pretraining [28] methods for completeness.…”

Section: Nyuv2 Semantic Segmentationmentioning

confidence: 99%

Contrastive Multimodal Fusion with TupleInfoNCE

Liu¹,

Fan

Zhang

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper proposes a method for representation learning of multimodal data using contrastive losses. A traditional approach is to contrast different modalities to learn the information shared among them. However, that approach could fail to learn the complementary synergies between modalities that might be useful for downstream tasks. Another approach is to concatenate all the modalities into a tuple and then contrast positive and negative tuple correspondences. However, that approach could consider only the stronger modalities while ignoring the weaker ones. To address these issues, we propose a novel contrastive learning objective, TupleInfoNCE. It contrasts tuples based not only on positive and negative correspondences, but also by composing new negative tuples using modalities describing different scenes. Training with these additional negatives encourages the learning model to examine the correspondences among modalities in the same tuple, ensuring that weak modalities are not ignored. We provide a theoretical justification based on mutual-information for why this approach works, and we propose a sample optimization algorithm to generate positive and negative samples to maximize training efficacy. We find that TupleInfoNCE significantly outperforms previous state of the arts on three different downstream tasks.

show abstract

Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis

Cited by 7 publications

References 37 publications

What Makes Multi-modal Learning Better than Single (Provably)

What Makes Multi-modal Learning Better than Single (Provably)

Improving Multi-Modal Learning with Uni-Modal Teachers

Contrastive Multimodal Fusion with TupleInfoNCE

Contact Info

Product

Resources

About