Creation and evaluation of large keyphrase extraction collections with multiple opinions

Abstract

While several Automatic Keyphrase Extraction (AKE) techniques have been developed and analyzed, there is little consensus on the definition of the task and a lack of overview of the effectiveness of different techniques. Proper evaluation of keyphrase extraction requires large test collections with multiple opinions, currently not available for research. In this paper, we (i) present a set of test collections derived from various sources with multiple annotations (which we also refer to as opinions in the remained of the paper) for each document, (ii) systematically evaluate keyphrase extraction using several supervised and unsupervised AKE techniques, (iii) and experimentally analyze the effects of disagreement on AKE evaluation. Our newly created set of test collections spans different types of topical content from general news and magazines, and is annotated with multiple annotations per article by a large annotator panel. Our annotator study shows that for a given document there seems to be a large disagreement on the preferred keyphrases, suggesting the need for multiple opinions per document. A first systematic evaluation of ranking and classification of keyphrases using both unsupervised and supervised AKE techniques on the test collections shows a superior effectiveness of supervised models, even for a low annotation effort and with basic positional and frequency features, and highlights the importance of a suitable keyphrase candidate generation approach. We also study the influence of multiple opinions, training data and document length on evaluation of keyphrase extraction. Our new test collection for keyphrase extraction is one of the largest of its kind and will be made available to stimulate future work to improve reliable evaluation of new keyphrase extractors.

A review of keyphrase extraction

“…To sum up, the problem of subjectivity could be partially addressed using multiple annotators or treating supervised keyphrase extraction as Positive Unlabeled Learning (Chuang, Manning, & Heer, ; Sterckx et al, ). In this direction, Sterckx, Caragea, Demeester, and Develder () propose as solution the creation of new collections for the evaluation of keyphrase extraction methods, derived from various sources with multiple annotations. The golden set of keyphrases, which is utilized for evaluation reasons, also incorporates subjectivity issues.…”

A survey on different dimensions for graphical keyword extraction techniques

“…( 2020 ), Dutch Sterckx et al. ( 2018 ), Polish (Pak2018), Spanish Aquino and Lanzarini ( 2015 ), Portuguese Marujo et al. ( 2013 ), and French Bougouin et al.…”

“…The classifications discussed in this article are limited to the GKET. Language Independence Academic researchers are working on AKE from texual documents of different languages, for instance, Telugu Naidu et al (2018), Chinese Chen et al (2020a), Li et al (2017), English (KPTimes and JPTimes), German Kölbl et al (2020), Dutch Sterckx et al (2018), Polish (Pak2018), Spanish Aquino and Lanzarini (2015), Portuguese , and French Bougouin et al (2013). The structure of the word distribution in Chinese and English documents is well studied using GoW evolved from these documents and different patterns are observed.…”

Keyphrase extraction techniques

“…In this direction, Sterckx et al (2018) propose as solution the creation of new collections for the evaluation of keyphrase extraction methods, derived from various sources with multiple annotations. This, however, comes at the expense of additional annotation effort.…”

“…Furthermore, several approaches to the creation of such datasets have been proposed, in the past (Medelyan et al, 2009;Sterckx et al, 2018)…”