Refining Sample Embeddings with Relation Prototypes to Enhance Continual Relation Extraction

Abstract

Continual learning has gained increasing attention in recent years, thanks to its biological interpretation and efficiency in many realworld applications. As a typical task of continual learning, continual relation extraction (CRE) aims to extract relations between entities from texts, where the samples of different relations are delivered into the model continuously. Some previous works have proved that storing typical samples of old relations in memory can help the model keep a stable understanding of old relations and avoid forgetting them. However, most methods heavily depend on the memory size in that they simply replay these memorized samples in subsequent tasks. To fully utilize memorized samples, in this paper, we employ relation prototype to extract useful information of each relation. Specifically, the prototype embedding for a specific relation is computed based on memorized samples of this relation, which is collected by K-means algorithm. The prototypes of all observed relations at current learning stage are used to re-initialize a memory network to refine subsequent sample embeddings, which ensures the model's stable understanding on all observed relations when learning a new task. Compared with previous CRE models, our model utilizes the memory information sufficiently and efficiently, resulting in enhanced CRE performance. Our experiments show that the proposed model outperforms the state-of-the-art CRE models and has great advantage in avoiding catastrophic forgetting. The code and datasets have been released on https://github.com/fd2014cl/RP-CRE.

Consistent Representation Learning for Continual Relation Extraction

“…In order to mitigate catastrophic forgetting in continual relational extraction, episodic memory modules have been used in previous work (Wang et al, 2019;Han et al, 2020;Cui et al, 2021), to store small samples in historical tasks. Inspired by (Cui et al, 2021), we store several representative samples for each relation. Therefore, the episodic memory module for the observed relations in…”

“…In order to make the model not forget the relevant knowledge of the old task when it learns the new task, some samples need to be stored in M r . Inspired by (Han et al, 2020;Cui et al, 2021), we use k-means to cluster each relation, where the number of clusters is the number of samples that need to be stored for each class. Then, the relation representation closest to the center is selected and stored in memory for each cluster.…”

“…To solve this situation, continual relation extraction (CRE) is introduced (Wang et al, 2019;Han et al, 2020;Wu et al, 2021;Cui et al, 2021). Compared with traditional relation extraction, CRE aims to help the model learn new relations while maintaining accurate classification of old ones.…”

“…Although these methods have been verified in simple image classification tasks, previous works have proved that memory-based methods are the most effective in natural language processing applications (Wang et al, 2019;d'Autume et al, 2019). In recent years, the memory-based continual relation extraction model has made significant progress in alleviating the problem of catastrophic forgetting (Han et al, 2020;Wu et al, 2021;Cui et al, 2021). Wang et al (2019) proposes a mechanism for embedding sentence alignment in memory maintenance to en-sure the stability of the embedding space.…”

“…Although it is possible to analyze the characteristics of each subtask and the performance of the corresponding model, it still fails to make full use of the saved sample information. Cui et al (2021) introduce an attention network to refine the prototype to better recover the interruption of the embedded space. However, this method will produce a bias in the classification of the old task as the new task continues to learn the classifier, which will affect the performance of the old task.…”