TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task

Abstract

TACRED (Zhang et al., 2017) is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE). But, even with recent advances in unsupervised pretraining and knowledge enhanced neural RE, models still show a high error rate. In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? And how do crowd annotations, dataset, and models contribute to this error rate? To answer these questions, we first validate the most challenging 5K examples in the development and test sets using trained annotators. We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled. On the relabeled test set the average F1 score of a large baseline model set improves from 62.1 to 70.1. After validation, we analyze misclassifications on the challenging instances, categorize them into linguistically motivated error groups, and verify the resulting error hypotheses on three state-of-the-art RE models. We show that two groups of ambiguous relations are responsible for most of the remaining errors and that models may adopt shallow heuristics on the dataset when entities are not masked.

Entities as Experts: Sparse Memory Access with Entity Supervision

“…We further demonstrate that EAE's learned entity representations are better than the pre-trained embeddings used by Zhang et al (2019); at knowledge probing tasks and the TA-CRED relation extraction task (Zhang et al, 2017;Alt et al, 2020). We show that training EAE to focus on entities is better than imbuing a similarsized network with an unconstrained memory store, and explain how EAE can outperform much larger sequence models while only accessing a small proportion of its parameters at inference time.…”

“…To determine whether this has a significant effect on a model's ability to model entity-entity relations, we evaluate EAE on TACRED (Zhang et al, 2017) dataset introduced by (Alt et al, 2020). 9 Table 7 shows that EAE outperforms KNOWBERT on the revised and weighted splits introduced by (Alt et al, 2020), although it slightly under-performs on the original setting. 10 This result indicates that EAE, without explicitly entity-entity attention, can capture relations between entities effectively.…”

Integrating Lexical Information into Entity Neighbourhood Representations for Relation Prediction

“…: (Cohen et al, 2020;Wang et al, 2019;Peters et al, 2019) 2 ) including the creation of many annotated data sets (e.g. : (Zhang et al, 2017;Alt et al, 2020;Mesquita et al, 2019;Elsahar et al, 2019)). These tasks consider only the recognition of knowledge directly expressed in individual texts, whereas we seek to utilise the combined knowledge from both a collection of texts and a knowledge base, allowing implicit and automatic association between expressions in texts and knowledge base relations and inference of propositions not directly expressed in individual texts.…”

Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data

“…Additional details in the supplementary material.6 We chose to perform binary annotation, as we find it makes the annotation process faster and more accurate. As demonstrated byAlt et al (2020), multi-class relation labeling by crowd-workers lead to frequent annotation errors. We observed the same phenomena also with non-crowd workers.…”