Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization

Abstract

Transcripts of natural, multi-person meetings differ significantly from documents like news articles, which can make Natural Language Generation models generate unfocused summaries. We develop an abstractive meeting summarizer from both videos and audios of meeting recordings. Specifically, we propose a multi-modal hierarchical attention mechanism across three levels: topic segment, utterance and word. To narrow down the focus into topically-relevant segments, we jointly model topic segmentation and summarization. In addition to traditional textual features, we introduce new multi-modal features derived from visual focus of attention, based on the assumption that an utterance is more important if its speaker receives more attention. Experiments show that our model significantly outperforms the state-of-the-art with both BLEU and ROUGE measures.

Abstractive Spoken Document Summarization Using Hierarchical Model with Multi-Stage Attention Diversity Optimization

“…Additional gains can be obtained with transfer learning (TL) where the model was initialized using the CNN/DailyMail data. The hierarchical models achieve higher ROUGE scores, consistent with [12]. Table 2: ROUGE F1 on the AMI test set -Baseline Models.…”

“…This relatively small drop is likely because even at 30% WER, the sentence/utterance embedding similarity between a manual source and an ASR source is about 0.70-0.85% [33,34]. This system achieves higher all ROUGE measures than extractive method CoreRank [6], and when compared to abstractive method Top-icSeg (without visual signals) [12] our system achieves higher ROUGE-2 and ROUGE-L although lower ROUGE-1. Furthermore, when using transcripts with lower WER (A2), ROUGE scores are closer to those obtained from the manual transcripts, and yield higher ROUGE-2 and ROUGE-L scores than the state-of-the-art multi-modal TopicSeg+VFOA.…”

“…,200 words, on average per meeting, and each summary contains 10 sentences of up to 300 words. Two sets of automatically derived transcripts (test set), from ASR systems using the manual segmentation, are used for testing our final model: • ASR1 -AMI release v1.5 which is publicly available and also used in [6,12]. The word error rate (WER) is 36%.…”

“…However, when it comes to spoken documents such as meeting transcripts, the input source is typically longer, less grammatical, and contains less-structured utterances rather than well-constructed sentences. Consequently, standard sequenceto-sequence neural models with attention mechanism [11] that take a long sequence of tokens have been shown to be less effective than hierarchical models for this task [12,13].…”

“…Although hierarchical models [15] do not outperform nonhierarchical models [8,16] on CNN/DailyMail, the hierarchical models are more suitable for meeting summarization [12,13]. Thus, in this work, we focus on hierarchical models.…”