Vector Space Modeling based Evaluation of Automatically Generated Text Summaries

Abstract

Evaluating automatically generated summaries is not an effortless task. Despite the fact that significant advances have been made in this context during the last two decades, it still remains a challenging resaerch problem. In this paper, we present VSMbM; a new metric for automatically generated text summaries evaluation. VSMbM is based on vector space modelling. It gives insights on to which extent retention and fidelity are met in the generated summaries. Three variants of the proposed metric, namely PCA-VSMbM, ISOMAP-VSMbM and tSNE-VSMbM are tested and compared to Recall-Oriented Understudy for Gisting Evaluation (ROUGE): a standard metric used to evaluate automatically generated summaries. Conducted experiments on the Timeline17 dataset show that VSMbM scores are highly correlated to the state-of-the-art Rouge ones.