Using Generative Pretrained Transformer-3 Models for Russian News Clustering and Title Generation tasks

Abstract

The paper presents a methodology for news clustering and news headline generation based on the zero-shot approach and minimal tuning of the RuGPT-3 architecture (Generative Pretrained Transformer 3 for Russian). The solution is presented in a competition for news clustering, headline selection and generation. The following approaches are described: 1) zero-shot unsupervised classification based on pairwise news perplexity: the method requires no training or model fine-tuning and yields 0.7 F1-measure. 2) fine-tuning: news headlines generation with the best result 0.292 ROUGE and 0.596 BLEU.