In Media Res: A Corpus for Evaluating Named Entity Linking with Creative Works

Abstract

Annotation styles express guidelines that direct human annotators by explicitly stating the rules to follow when creating gold standard annotations of text corpora. These guidelines not only shape the gold standards they help create, but also influence the training and evaluation of Named Entity Linking (NEL) tools, since different annotation styles correspond to divergent views on the entities present in a document. Such divergence is particularly relevant for texts from the media domain containing references to creative works. This paper presents a corpus of 1000 annotated documents from sources such as Wikipedia, TVTropes and WikiNews that are organized in ten partitions. Each document contains multiple gold standard annotations representing various annotation styles. The corpus is used to evaluate a series of Named Entity Linking tools in order to understand the impact of the differences in annotation styles on the reported accuracy when processing highly ambiguous entities such as names of creative works. Relaxed annotation guidelines that include overlap styles, for instance, lead to better results across all tools.

A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems

“…Apart from that, they do not provide guidelines or suggestions that target the annotation process. Brasoveanu et al (2018) argue that an in-depth qualitative analysis of entity linking errors is necessary in order to efficiently improve entity linking systems. They categorize EL errors into five categories: knowledge base errors, dataset errors, annotator errors, NIL clustering errors and evaluation errors.…”

Data-driven personalisation of television content: a survey

“…and specific training of NER/NEL systems is needed for this case. In the ReTV project, a customisation of the Recogynze tool performed significantly better than the previously mentioned "off the shelf" systems [211]. These annotation tools are generically focused on the identification of the occurrence of entities in text, rather than the specific use case of describing the content of TV programming for personalisation services.…”