Indosum: A New Benchmark Dataset for Indonesian Text Summarization

Abstract

Automatic text summarization is generally considered as a challenging task in the NLP community. One of the challenges is the publicly available and large dataset that is relatively rare and difficult to construct. The problem is even worse for low-resource languages such as Indonesian.In this paper, we present INDOSUM, a new benchmark dataset for Indonesian text summarization. The dataset consists of news articles and manually constructed summaries. Notably, the dataset is almost 200x larger than the previous Indonesian summarization dataset of the same domain. We evaluated various extractive summarization approaches and obtained encouraging results which demonstrate the usefulness of the dataset and provide baselines for future research. The code and the dataset are available online under permissive licenses.

IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP

“…Because the sentiment analysis data is low-resource and imbalanced, we use stratified 5-fold cross-validation, and evaluate based on F1 score. For summarization, on the other hand, we use the canonical splits provided by Kurniawan and Louvan (2018), and evaluate the resulting summary with ROUGE (F1) (Lin, 2004) in the form of three different metrics: R1, R2, and RL.…”

“…Aristoteles et al (2012) deployed a genetic algorithm over a 200-document summarization dataset, and Gunawan et al (2017) performed unsupervised summarization over 3,075 news articles. As an attempt to create a standardized corpus, Koto (2016) released a 300-document chat summarization dataset, and Kurniawan and Louvan (2018) released the IndoSum 19K document-summary dataset. At the time we carried out this work, 20 IndoSum was the largest Indonesian summarization corpus in the news domain, manually constructed from CNN Indonesia 21 and Kumparan 22 documents.…”

“…IndoSum is a single-document summarization dataset where each article has one abstractive summary. Kurniawan and Louvan (2018) released IndoSum together with the ORACLE -a set of extractive summaries generated automatically by maximizing ROUGE score between sentences of the article and its abstractive summary. We include IndoSum as the summarization dataset in INDOLEM, and evaluate the performance of extractive summarization in this paper.…”

“…For extractive summarization baselines, we use the models of Kurniawan and Louvan (2018) and Cheng and Lapata (2016) as baselines. Kurniawan and Louvan (2018) propose a sentence tagging approach based on a hidden Markov model, while Cheng and Lapata (2016) Table 2: Comparison of baselines and BERT-based models for all INDOLEM tasks. All listed models were implemented and run by the authors, except for those marked with " †" where the results are sourced from the original paper.…”

Abstract Text Summarization: A Low Resource Challenge

“…Further improvements are introduced to the baseline model by using the pointer generator network and coverage mechanism using reinforcement learning based training procedure (See et al, 2017;Paulus et al, 2017). There is an inherent limitation to natural language processing tasks such as text summarization for resource-poor and morphological complex languages owing to a shortage of quality linguistic data available (Kurniawan and Louvan, 2018). The use of synthetic data along with the real data is one of the popular approaches followed in machine translation domain for the low resource conditions to improve the translation quality (Bojar and Tamchyna, 2011;Hoang et al, 2018;Chinea-Rıos et al, 2017).…”