ERNIE at SemEval-2020 Task 10: Learning Word Emphasis Selection by Pre-trained Language Model

Abstract

This paper describes the system designed by ERNIE Team which achieved the first place in SemEval-2020 Task 10: Emphasis Selection For Written Text in Visual Media. Given a sentence, we are asked to find out the most important words as the suggestion for automated design. We leverage the unsupervised pre-training model and finetune these models on our task. After our investigation, we found that the following models achieved an excellent performance in this task: ERNIE 2.0, XLM-ROBERTA, ROBERTA and ALBERT. We combine a pointwise regression loss and a pairwise ranking loss which is more close to the final M atch m metric to finetune our models. And we also find that additional feature engineering and data augmentation can help improve the performance. Our best model achieves the highest score of 0.823 and ranks first for all kinds of metrics.

PSED: A Dataset for Selecting Emphasis in Presentation Slides

“…For training and evaluation purposes, an emphasis selection dataset with social media short texts from Adobe Spark and publicly available quotes was introduced. Top-performing teams, ERNIE (Huang et al 2020), Hitachi (Morio et al 2020), and IITK (Singhal et al 2020) were able to achieve the first, second, and third places respectively by utilizing rich contextualized pre-trained language models such as ERNIE 2.0 , XLMRoBERTa , XLNet (Yang et al 2019), andT5 (Raffel et al 2019). In this study, we focus on a new domain, presentation slide, which emphasizes the importance of utilizing visual tools to convey a more effective presentation.…”

EmpLite: A Lightweight Sequence Labeling Model for Emphasis Selection of Short Texts

“…One drawback of using label distribution learning is the requirement of annotations, which are not readily available in most datasets. Pre-trained language model has also been used to achieve emphasis selection (Huang et al, 2020). Singhal et al (Singhal et al, 2020) achieves significantly good performance with (a) Bi-LSTM + Attention approach, and (b) Transformers approach.…”