D-MmT: A concise decoder-only multi-modal transformer for abstractive summarization in videos

MCR: Multilayer cross‐fusion with reconstructor for multimodal abstractive summarisation

“…HA [1] is a baseline multiencoder‐decoder model with a hierarchical attention strategy in the decoder part; MFFG [3] is a multi‐stage fusion architecture to fuse multi‐source data, which suppresses the flow of multimodal noise via a forget gate module; D ‐ MmT [2] is a decoder‐only multimodal transformer framework for video‐containing the MAS task.…”

“…Shang et al [27] introduced a novel short-term order-sensitive attention mechanism to leverage the time clue inside video frames. Liu et al [2] proposed reducing model parameters with a decoder-only multimodal transformer which combines the source inputs and target summary in the shared feature space.…”

“…Liu et al. [2] proposed reducing model parameters with a decoder‐only multimodal transformer which combines the source inputs and target summary in the shared feature space.…”

“…Existing approaches have obtained promising results, and current methods can be roughly divided into two categories. As shown in Figure 1a, some works [1,2] applied specific encoders to process source inputs and obtain single-modal embeddings. Then they used a summary generation decoder to fuse multimodal information.…”

Multimodal Abstractive Summarization using bidirectional encoder representations from transformers with attention mechanism

“…Liu et al [ 22 ] have introduced a Decoder-only Multimodal Transformer (D-MmT), which is modified from the structure of the decoder by including the in-out multimodal decoder. Moreover, Cascaded Cross-Modal interaction (CXMI) creates the joint fusion representation among the modalities.…”