Overview of CLEF HIPE 2020: Named Entity Recognition and Linking on Historical Newspapers

Abstract

This paper presents an overview of the first edition of HIPE (Identifying Historical People, Places and other Entities), a pioneering shared task dedicated to the evaluation of named entity processing on historical newspapers in French, German and English. Since its introduction some twenty years ago, named entity (NE) processing has become an essential component of virtually any text mining application and has undergone major changes. Recently, two main trends characterise its developments: the adoption of deep learning architectures and the consideration of textual material originating from historical and cultural heritage collections. While the former opens up new opportunities, the latter introduces new challenges with heterogeneous, historical and noisy inputs. In this context, the objective of HIPE, run as part of the CLEF 2020 conference, is threefold: strengthening the robustness of existing approaches on non-standard inputs, enabling performance comparison of NE processing on historical texts, and, in the long run, fostering efficient semantic indexing of historical documents. Tasks, corpora, and results of 13 participating teams are presented.

A survey of OCR evaluation tools and metrics

“…They observed a drop in accuracy of the NER from 90% to 60% when the word error rate was increased from 1% to 7% and the character error rate was increased from 8% to 20%, respectively. An evaluation of the submitted systems of the Shared Task HIPE came to similar results regarding the dependency of NER results and OCR quality for historical texts [10]. In addition to recognizing named entities, linking them to knowledge bases (named entity linking, NEL) can be used to disambiguate ambiguous proper names.…”

Survey on English Entity Linking on Wikidata: Datasets and approaches

“…Or some NIL entities that do not exist in other KGs could exist in Wikidata. Eleven datasets [16,23,24,27,29,33,46,56,69,80] were found for which Wikidata identifiers were available from the start. In the following the datasets are separated by their domain.…”

“…It is an annotated EL dataset based on the KORE50 dataset, a manually annotated subset of the AIDA-CoNLL corpus. The CLEF HIPE 2020 [29] is a dataset based on historical newspapers in English, French and German. Only the English dataset will be analyzed in the following.…”

“…Kensho Derived Wikimedia Dataset [56] 1 [ 86] CLEF HIPE 2020 [29] 3 [ 15,65,89] Mewsli-9 [16] 1 [ 16] TweekiData [46] 1 [ 46] TweekiGold [46] 1 [ 46] Table 9 Ambiguity of mentions (existence of a match does not correspond to a correct match), NYT2018 dataset was not available and LC-QuAD Wikimedia Dataset and the Mewsli-9 training dataset have the largest percentage of exact matches for labels. While TweekiGold is quite ambiguous, deciding on the most prominent entity appears to produce good EL results.…”

“…Except the Mewsli-9 [16] and CLEF HIPE 2020 [29] datasets, none of the others take any specific characteristics of Wikidata into account. The two exceptions focus on multilinguality and rely therefore directly on the languageagnostic nature of Wikidata.…”