Packing the meeting summarization knapsack

Abstract

Despite considerable work in automatic meeting summarization over the last few years, comparing results remains difficult due to varied task conditions and evaluations. To address this issue, we present a method for determining the best possible extractive summary given an evaluation metric like ROUGE. Our oracle system is based on a knapsack-packing framework, and though NP-Hard, can be solved nearly optimally by a genetic algorithm. To frame new research results in a meaningful context, we suggest presenting our oracle results alongside two simple baselines. We show oracle and baseline results for a variety of evaluation scenarios that have recently appeared in this field.

Unsupervised Abstractive Meeting Summarization with Multi-Sentence Compression and Budgeted Submodular Maximization

“…We conducted experiments on the widely-used AMI (McCowan et al, 2005) and ICSI (Janin et al, 2003) benchmark datasets. We used the traditional test sets of 20 and 6 meetings respectively for the AMI and ICSI corpora (Riedhammer et al, 2008). Each meeting in the AMI test set is associated with a human abstractive summary of 290 words on average, whereas each meeting in the ICSI test set is associated with 3 human abstractive summaries of respective average sizes 220, 220 and 670 words.…”

“…• Random and Longest Greedy are basic baselines recommended by (Riedhammer et al, 2008) • Oracle is the same as the random baseline, but uses the human extractive summaries as input.…”

“…• Random. A basic baseline recommended by (Riedhammer et al, 2008) to ease cross-study comparison. This system randomly selects utterances without replacement from the transcription until the budget is violated.…”

“…• Longest Greedy. A basic baseline recommended by (Riedhammer et al, 2008) to ease cross-study comparison. The longest remaining utterance is selected at each step from the transcription until the summary size constraint is satisfied.…”

Enumeration of Extractive Oracle Summaries

“…To describe the difference between the ROUGE n scores of oracle and system summaries in multiple document summarization tasks, Riedhammer et al (2008) proposed an approximate algorithm with a genetic algorithm (GA) to find oracle summaries. Moen et al (2014) utilized a greedy algorithm for the same purpose.…”