WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization

Ladhak, Faisal; Durmus, Esin; Cardie, Claire; McKeown, Kathleen

doi:10.18653/v1/2020.findings-emnlp.360

Cited by 105 publications

(107 citation statements)

References 32 publications

Supporting

Mentioning

103

Contrasting

Unclassified

Order By: Relevance

“…The human evaluation results correlate with automated evaluation as shown in Tables 1 and 2. Ladhak et al (2020) reported cross-lingual ATS score with same data for four different languages. The R-L score for four languages are 34.…”

Section: Abstractive Text Summarization (Ats)mentioning

confidence: 97%

“…In Abstractive Text Summarization (ATS), we aim to generate grammatically coherent, semantically correct and abstractive summary given an input document. We use recently released WikiLingua (Ladhak et al, 2020) cross-lingual abstractive summarization dataset containing data in 18 languages. Prior splits are not available for this dataset.…”

Section: Abstractive Text Summarization (Ats)mentioning

confidence: 99%

See 1 more Smart Citation

ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation

Maurya¹,

Desarkar²,

Kano³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Despite the recent advancement in NLP research, cross-lingual transfer for natural language generation is relatively understudied. In this work, we transfer supervision from high resource language (HRL) to multiple lowresource languages (LRLs) for natural language generation (NLG). We consider four NLG tasks (text summarization, question generation, news headline generation, and distractor generation) and three syntactically diverse languages, i.e., English, Hindi, and Japanese. We propose an unsupervised crosslingual language generation framework (called ZmBART) that does not use any parallel or pseudo-parallel/back-translated data. In this framework, we further pre-train mBART sequence-to-sequence denoising auto-encoder model with an auxiliary task using monolingual data of three languages. The objective function of the auxiliary task is close to the target tasks which enriches the multi-lingual latent representation of mBART and provides good initialization for target tasks. Then, this model is fine-tuned with task-specific supervised English data and directly evaluated with low-resource languages in the Zero-shot setting. To overcome catastrophic forgetting and spurious correlation issues, we applied freezing model component and data argumentation approaches respectively. This simple modeling approach gave us promising results. We experimented with few-shot training (with 1000 supervised data-points) which boosted the model performance further. We performed several ablations and cross-lingual transferability analysis to demonstrate the robustness of ZmBART.

show abstract

Section: Abstractive Text Summarization (Ats)mentioning

confidence: 97%

Section: Abstractive Text Summarization (Ats)mentioning

confidence: 99%

ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation

Maurya¹,

Desarkar²,

Kano³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

show abstract

“…The summarization landscape can be roughly divided into three primary summary-forms: (1) Single sentence (Napoles et al, 2012;Grusky et al, 2018;Narayan et al, 2018;Kim et al, 2019) -summarize the document in a single sentence; (2) Highlights (Hermann et al, 2015;Koupaee and Wang, 2018;Ladhak et al, 2020) -a summary in the form of bullets listing the key points in the text; (3) Coherent summary (Sharma et al, 2019;Cohan et al, 2018) -short coherent paragraphs describing the salient information. The summarization datasets from the news domain, which are commonly used for human evaluation, include summaries in the form of highlights or single-sentence summaries.…”

Section: Related Workmentioning

confidence: 99%

“…PubMed, arXiv, and BigPatent (Cohan et al, 2018;Sharma et al, 2019) provide a summary in the form of coherent paragraphs (i.e., each sentence flows smoothly into the next). In contrast, other summarization datasets (Hermann et al, 2015;Grusky et al, 2018;Koupaee and Wang, 2018;Ladhak et al, 2020) offer a summary in the form of a key points list (i.e., highlights). In this paper, we focus on coherent paragraph summarization datasets.…”

Section: Introductionmentioning

confidence: 99%

“…The WikiSum documents are written in simple English, and the summaries provide "nonobvious tips that mimic the advice a knowledgeable, empathetic friend might give." 3 Unlike previous WikiHow summarization (Koupaee and Wang, 2018;Ladhak et al, 2020) from the news domain, the summaries of WikiSum are in the form of a coherent paragraph written by the document authors (examples in Figure 1). Moreover, in contrast to other coherent-paragraph summarization datasets from the academic domain, WikiSum is written using simple English.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation

Cohen¹,

Kalinsky²,

Ziser³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Recent works have made significant advances on summarization tasks, facilitated by summarization datasets. Several existing datasets have the form of coherent-paragraph summaries. However, these datasets were curated from academic documents written for experts, making the essential step of assessing the summarization output through human-evaluation very demanding.To overcome these limitations, we present a dataset 1 based on article summaries appearing on the WikiHow website, composed of howto articles and coherent-paragraph summaries written in plain language. We compare our dataset attributes to existing ones, including readability and world-knowledge, showing our dataset makes human evaluation significantly more manageable and effective. A human evaluation conducted on PubMed and the proposed dataset reinforces our findings.

show abstract

To Adapt or to Fine-Tune: A Case Study on Abstractive Summarization

Zheng

Chen

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Recent advances in the field of abstractive summarization leverage pre-trained language models rather than train a model from scratch. However, such models are sluggish to train and accompanied by a massive overhead. Researchers have proposed a few lightweight alternatives such as smaller adapters to mitigate the drawbacks. Nonetheless, it remains uncertain whether using adapters benefits the task of summarization, in terms of improved efficiency without an unpleasant sacrifice in performance. In this work, we carry out multifaceted investigations on fine-tuning and adapters for summarization tasks with varying complexity: language, domain, and task transfer. In our experiments, fine-tuning a pre-trained language model generally attains a better performance than using adapters; the performance gap positively correlates with the amount of training data used. Notably, adapters exceed fine-tuning under extremely low-resource conditions. We further provide insights on multilinguality, model convergence, and robustness, hoping to shed light on the pragmatic choice of fine-tuning or adapters in abstractive summarization.

show abstract

WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization

Cited by 105 publications

References 32 publications

ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation

ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation

WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation

To Adapt or to Fine-Tune: A Case Study on Abstractive Summarization

Contact Info

Product

Resources

About