ROUGE-SS: A New ROUGE Variant for Evaluation of Text Summarization

Abstract

Evaluation is the systematic collection and analysis of data to make judgments about a software system’s value, worth or quality. The authenticity or accuracy of any software system is based on evaluation techniques. If the evaluation technique covers more features for evaluating a software system, then it will be beneficial for determining the proposed system’s accuracy, validity, and reliability. The ROUGE score metrics and their variants are utilized for evaluating text summarization models. While ROUGE metrics are suitable for extractive approaches, they are inadequate for abstractive approaches as it rely on exact word matching. Therefore, we have proposed a new variant of the ROUGE metric called ROUGE-SS, which also considers word’s synonyms in addition to exact matches. Our experiments have shown that ROUGE-SS is more effective than other variants of ROUGE scores. The f1-score of proposed ROUGE-SS metric is increase by an average of 8.8%.