SUMMAC: a text summarization evaluation

Abstract

The TIPSTER Text Summarization Evaluation (SUMMAC) has developed several new extrinsic and intrinsic methods for evaluating summaries. It has established definitively that automatic text summarization is very effective in relevance assessment tasks on news articles. Summaries as short as 17% of full text length sped up decision-making by almost a factor of 2 with no statistically significant degradation in accuracy. Analysis of feedback forms filled in after each decision indicated that the intelligibility of present-day machine-generated summaries is high. Systems that performed most accurately in the production of indicative and informative topic-related summaries used term frequency and co-occurrence statistics, and vocabulary overlap comparisons between text passages. However, in the absence of a topic, these statistical methods do not appear to provide any additional leverage: in the case of generic summaries, the systems were indistinguishable in accuracy. The paper discusses some of the tradeoffs and challenges faced by the evaluation, and also lists some of the lessons learned, impacts, and possible future directions. The evaluation methods used in the SUMMAC evaluation are of interest to both summarization evaluation as well as evaluation of other 'output-related' NLP technologies, where there may be many potentially acceptable outputs, with no automatic way to compare them.

Syntactic sentence compression in the biomedical domain: facilitating access to related articles

“…However, quantifying performance in terms of decision-making accuracy on compressed titles (as compared to the full titles) is informative because it illustrates how summarization techniques assist the user's end task, that of knowledge exploration and information gathering. In general, we believe that information presentation issues provide a general framework for task-based evaluation of summarization systems; see also, (Mani et al 2002;Dorr et al 2005) for similar setups.…”

“…Developing realistic usage scenarios is challenging, but often the ''goodness'' of a summary can only be meaningfully operationalized in its ''usefulness'' for a particular task. One might, for example, measure how summaries impact question answering (Morris et al 1992;Mani et al 2002) or relevance judgments (Dorr et al 2005). One possible hypothesis is that summaries allow users to make quicker decisions (since they have to read less), without compromising the quality of those decisions.…”

“…This is similar to the consistency test employed in the TIPSTER SUMMAC evaluations (Mani et al 2002). Results can be analyzed in terms of the contingency table shown in Table 7, from which we can compute standard aggregate statistics:…”

Movie Script Summarization as Graph-based Scene Extraction

“…In addition, we also evaluated the quality of the extracted scenes as perceived by humans, which is necessary, given the approximate nature of our gold standard. We adopted a question-answering (Q&A) evaluation paradigm which has been used previously to evaluate summaries and document compression (Morris et al, 1992;Mani et al, 2002;Clarke and Lapata, 2010). Under the assumption that the summary is to function as a replacement for the full script, we can measure the extent to which it can be used to find answers to questions which have been derived from the entire script and are representative of its core content.…”

A Survey to Text Summarization Methods for Turkish

“…The abstract may provide an introductory overview of its topic or argument for readers to whom the document is of marginal interest, and make a reading of the full document unnecessary" [2]. Also summaries as short as 17% of full text length sped up decision-making by almost a factor of 2 with no statistically significant degradation in accuracy [3].…”