Low Rank Fusion based Transformers for Multimodal Sequences

Sahay, Saurav; Okur, Eda; Kumar, Samir; Nachman, Lama

doi:10.48550/arxiv.2007.02038

Cited by 9 publications

(19 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experimental results show that the proposed method achieves a comparable performance level to PMR (Lv et al 2021) on different metrics for the three datasets. Compared with LMF-MulT (Sahay et al 2020), which uses six transformer encoders, we achieve better performance on different datasets using half of the transformer encoders.…”

Section: Comparison With the State-of-the-artsmentioning

confidence: 99%

“…Tsai et al (2019a) proposed a multimodal transformer (MulT) to learn intermodal correlations using a cross-modal attention mechanism. Sahay et al (2020) proposed low rank fusion based transformers (LMT-MULT) to design LMF units for efficient modal feature fusion based on previous work. Lv et al (2021) proposed progressive modality reinforcement (PMR) method.…”

Section: Related Workmentioning

confidence: 99%

“…We compare the proposed approach with the existing stateof-the-art methods, including Early Fusion LSTM (EF-LSTM), Late Fusion LSTM (LF-LSTM), Multimodal Factorization Model (MFM) (Tsai et al 2019b), Graph-MFN (GMFN), Recurrent Attended Variation Embedding Network (RAVEN) (Wang et al 2019), Multimodal Cyclic Translation Network (MCTN) (Pham et al 2019), Multimodal Transformer (MulT) (Tsai et al 2019a), Low Rank Fusion based Transformers (LMF-MulT) (Sahay et al 2020), Modality-Invariant and -Specific Representations (MISA) (Hazarika, Zimmermann, and Poria 2020), Pro-gressive Modality Reinforcement (PMR) (Lv et al 2021). Among these methods, LF-LSTM, MulT, LMF-MulT, and PMR can be directly applied the unaligned setting.…”

Section: Comparison With the State-of-the-artsmentioning

confidence: 99%

“…• With the exception of MulT (Tsai et al 2019a), LMF-MulT (Sahay et al 2020), and PMR (Lv et al 2021), most of the models perform poorly in the unaligned setting because they do not take into account the interactions between the modalities. In addition, the outstanding performance of MISA (Hazarika, Zimmermann, and Poria 2020) PMR on the CMU-MOSEI dataset reaches 15.9 M and 2.15 M, respectively, while our proposed method uses only 0.41 M. For MISA, the number of parameters is equivalent to 38 times that of our proposed method, while PMR is equivalent to as much as 6 times.…”

Section: Comparison With the State-of-the-artsmentioning

confidence: 99%

“…In the previous works (Sahay et al 2020;Rahman et al 2020;Hazarika, Zimmermann, and Poria 2020;Yu et al 2021;Dai et al 2021a), Transformers (Vaswani et al 2017) are mostly used for unaligned multimodal emotion recognition. Typically, Tsai et al (2019a) proposed the Multimodal Transformer (MulT) method to fuse information from different modalities in unaligned sequences without explicitly aligning the data.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

Fu¹,

Liu²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Learning modality-fused representations and processing unaligned multimodal sequences are meaningful and challenging in multimodal emotion recognition. Existing approaches use directional pairwise attention or a message hub to fuse language, visual, and audio modalities. However, those approaches introduce information redundancy when fusing features and are inefficient without considering the complementarity of modalities. In this paper, we propose an efficient neural network to learn modality-fused representations with CB-Transformer (LMR-CBT) for multimodal emotion recognition from unaligned multimodal sequences. Specifically, we first perform feature extraction for the three modalities respectively to obtain the local structure of the sequences. Then, we design a novel transformer with cross-modal blocks (CB-Transformer) that enables complementary learning of different modalities, mainly divided into local temporal learning, cross-modal feature fusion and global self-attention representations. In addition, we splice the fused features with the original features to classify the emotions of the sequences. Finally, we conduct word-aligned and unaligned experiments on three challenging datasets, IEMOCAP, CMU-MOSI, and CMU-MOSEI. The experimental results show the superiority and efficiency of our proposed method in both settings. Compared with the mainstream methods, our approach reaches the state-of-the-art with a minimum number of parameters.

show abstract

Section: Comparison With the State-of-the-artsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Comparison With the State-of-the-artsmentioning

confidence: 99%

Section: Comparison With the State-of-the-artsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

Fu¹,

Liu²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

DeFACT in ManuVerse for Parallel Manufacturing: Foundation Models and Parallel Workers in Smart Factories

Yang

Chen³

et al. 2023

IEEE Trans. Syst. Man Cybern, Syst.

View full text Add to dashboard Cite

In cyber-physical-social systems, smart manufacturing has to overcome challenges, such as uncertainty, diversity, complexity in modeling, long-delayed responses to market changes, and human engineer dependency. DeFACT is a framework of parallel manufacturing in ManuVerse where the Decentralized Autonomous Organization-based interactions between parallel workers consisting of robotic, digital, and human workers are elaborated to transform from professional division to real-virtual division. In DeFACT, human workers are only responsible for 5% physical and mental work that is complex and creative, and the robotic and digital workers can take care of the rest. The perceptual and cognitive intelligence of digital workers are intensified by a manufacturing foundation model (MF-PC), where calibration and certification (C&C), and verification and validation (V&V) guarantee not only the accuracy of task models, but also the interpretability and controllability of feature learning. As a case study, the workflow of customized shoes of SANBODY Technology Company is illustrated to show how DeFACT breaks the time and space constraints, avoids production waste caused by aesthetic discrepancies with consumers, and truly realizes flexible manufacturing.

show abstract

Vision transformer quantization with multi-step knowledge distillation

Ranjan,

Savakis

2024

Signal Processing, Sensor/Information Fusion, and Target Recognition XXXIII

View full text Add to dashboard Cite

Vision Transformers (ViTs) have demonstrated remarkable performance in various visual tasks, but they suffer from expensive computational and memory challenges, which hinder their practical application in the real world. Model quantization methods reduce the model computation and memory requirements through low-bit representations. Knowledge distillation is used to guide the quantized student network to imitate the performance of its full-precision counterpart teacher network. However, for ultra-low bit quantization, the student networks experience a noticeable performance drop. This is primarily due to the limited learning capacity of the smaller network to capture the knowledge of the full-precision teacher, especially when the representation gaps between the student and the teacher networks are significant. In this paper, we introduce a multi-step knowledge distillation approach, utilizing intermediate-quantized networks with varying bit precision. This multi-step knowledge distillation approach enables an ultra-low bit quantized student network to effectively bridge the gap with the teacher network by gradually reducing the model's bit representation. We progressively teach each TA network to learn by distilling the knowledge from higher-bit quantized teacher networks from the previous step. The target student network learns from the combined knowledge of the teacher assistants and the full-precision teacher network, resulting in improved learning capacity even when faced with significant knowledge gaps. We evaluate our methods using the DeiT vision transformer for both ground level and aerial image classification tasks.

show abstract

Low Rank Fusion based Transformers for Multimodal Sequences

Cited by 9 publications

References 0 publications

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

DeFACT in ManuVerse for Parallel Manufacturing: Foundation Models and Parallel Workers in Smart Factories

Vision transformer quantization with multi-step knowledge distillation

Contact Info

Product

Resources

About