NASca and NASes: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish

Abstract

Most of the models proposed in the literature for abstractive summarization are generally suitable for the English language but not for other languages. Multilingual models were introduced to address that language constraint, but despite their applicability being broader than that of the monolingual models, their performance is typically lower, especially for minority languages like Catalan. In this paper, we present a monolingual model for abstractive summarization of textual content in the Catalan language. The model is a Transformer encoder-decoder which is pretrained and fine-tuned specifically for the Catalan language using a corpus of newspaper articles. In the pretraining phase, we introduced several self-supervised tasks to specialize the model on the summarization task and to increase the abstractivity of the generated summaries. To study the performance of our proposal in languages with higher resources than Catalan, we replicate the model and the experimentation for the Spanish language. The usual evaluation metrics, not only the most used ROUGE measure but also other more semantic ones such as BertScore, do not allow to correctly evaluate the abstractivity of the generated summaries. In this work, we also present a new metric, called content reordering, to evaluate one of the most common characteristics of abstractive summaries, the rearrangement of the original content. We carried out an exhaustive experimentation to compare the performance of the monolingual models proposed in this work with two of the most widely used multilingual models in text summarization, mBART and mT5. The experimentation results support the quality of our monolingual models, especially considering that the multilingual models were pretrained with many more resources than those used in our models. Likewise, it is shown that the pretraining tasks helped to increase the degree of abstractivity of the generated summaries. To our knowledge, this is the first work that explores a monolingual approach for abstractive summarization both in Catalan and Spanish.

Enhancing Text Summarization with a T5 Model and Bayesian Optimization

“…Many studies have carried out a combination of models in carrying out text summary tasks, as was done by Ahuir et al [43] who carried out text summaries using a combination of a transfer model that had been trained with the T5 model, another study conducted by Chouikhi and Alsuhaibani [44] who carried out text summaries with the T5 model. deep transformers that perform the hyperparameter combinations of the T5 model.…”

Current Approaches and Applications in Natural Language Processing

“…As an additional contribution, this time related to text summarization, is the work presented in [25]. It is a monolingual approach for abstractive summarization in Catalan and Spanish.…”

esT5s: A Spanish Model for Text Summarization

“…The state of the art for Spanish text abstractive summarization is the model published by Hasan et al [9] based on the mT5 model (this model will be referred as mT5Hasan) and NASES [10], both reporting the highest ROUGE scores. It is important to mention that mT5Hasan ROUGE scores are calculated with the XL-Sum dataset, while NASES scores are calculated with a dataset that is not public and authors do not point out any indication on the dataset used.…”