Semi-supervised disentangled framework for transferable named entity recognition

Zhang, Hao; Lv, Di; Li, Zijian; Cai, Ruichu; Wen, Wen; Xu, Boyan

doi:10.1016/j.neunet.2020.11.017

Cited by 14 publications

(9 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…GMI [28] brings mutual information into graph representation learning to alleviate the problem of lacking available supervision and avoid potential risks from unreliable labels. In addition, SSD [12] is a disentanglement framework, where mutual information is served as supervision signals for domain adaptation tasks. For DGRe, we utilize mutual information constraints similar to [12], but apply them into the disentanglement of the document representation for document ranking task.…”

Section: Other Related Techniquesmentioning

confidence: 99%

See 1 more Smart Citation

Disentangled Graph Recurrent Network for Document Ranking

et al. 2022

View full text Add to dashboard Cite

BERT-based ranking models are emerging for its superior natural language understanding ability. All word relations and representations in the concatenation of query and document are modeled in the self-attention matrix as latent knowledge. However, some latent knowledge has none or negative effect on the relevance prediction between query and document. We model the observable and unobservable confounding factors in a causal graph and perform do-query to predict the relevance label given an intervention over this graph. For the observed factors, we block the back door path by an adaptive masking method through the transformer layer and refine word representations over this disentangled word graph through the refinement layer. For the unobserved factors, we resolve the do-operation query from the front door path by decomposing word representations into query related and unrelated parts through the decomposition layer. Pairwise ranking loss is mainly used for the ad hoc document ranking task, triangle distance loss is introduced to both the transformer and refinement layers for more discriminative representations, and mutual information constraints are put on the decomposition layer. Experimental results on public benchmark datasets TREC Robust04 and WebTrack2009-12 show that DGRe outperforms state-of-the-art baselines more than 2% especially for short queries.

show abstract

Section: Other Related Techniquesmentioning

confidence: 99%

“…In addition, SSD [12] is a disentanglement framework, where mutual information is served as supervision signals for domain adaptation tasks. For DGRe, we utilize mutual information constraints similar to [12], but apply them into the disentanglement of the document representation for document ranking task.…”

Section: Other Related Techniquesmentioning

confidence: 99%

Disentangled Graph Recurrent Network for Document Ranking

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The most obviously explainable networks are those in which individual dimensions of the latent space more or less directly reflect or represent identifiable features of the inputs; in the case of images of faces, for example, this would occur when the value of a feature in one dimension varies smoothly with and thus can be seen to represent, an input feature such as hair color, the presence or type of spectacles, the presence or type of a moustache, and so on [23][24][25][26]. This is known as a disentangled representation (e.g., [27][28][29][30][31][32][33][34][35][36][37][38]). To this end, it is worth commenting that the ability to generate more or less realistic facial image structures using orthogonal features extracted from a database or collection of relevant objects that can be parametrized has been known for some decades [39][40][41][42].…”

Section: Introductionmentioning

confidence: 99%

FragNet, a Contrastive Learning-Based Transformer Model for Clustering, Interpreting, Visualizing, and Navigating Chemical Space

Shrivastava

Kell

2021

Molecules

View full text Add to dashboard Cite

The question of molecular similarity is core in cheminformatics and is usually assessed via a pairwise comparison based on vectors of properties or molecular fingerprints. We recently exploited variational autoencoders to embed 6M molecules in a chemical space, such that their (Euclidean) distance within the latent space so formed could be assessed within the framework of the entire molecular set. However, the standard objective function used did not seek to manipulate the latent space so as to cluster the molecules based on any perceived similarity. Using a set of some 160,000 molecules of biological relevance, we here bring together three modern elements of deep learning to create a novel and disentangled latent space, viz transformers, contrastive learning, and an embedded autoencoder. The effective dimensionality of the latent space was varied such that clear separation of individual types of molecules could be observed within individual dimensions of the latent space. The capacity of the network was such that many dimensions were not populated at all. As before, we assessed the utility of the representation by comparing clozapine with its near neighbors, and we also did the same for various antibiotics related to flucloxacillin. Transformers, especially when as here coupled with contrastive learning, effectively provide one-shot learning and lead to a successful and disentangled representation of molecular latent spaces that at once uses the entire training set in their construction while allowing “similar” molecules to cluster together in an effective and interpretable way.

show abstract

“…Based on the above idea, we propose the Temporally Evolving Aggregation (TEA in short) framework for sequential recommendation by aggregating the user behavior sequence as well as the dynamic user-item heterogeneous graph. Inspired by the sequence labeling in natural language processing [18], [19] to model the joint probability distribution), we adopt CRF to model the item decision sequence and estimate P (v t+1 |u i , H 1:t+1 ; v 1:t ). In order to alleviate the issue of the large item space, we use the pseudo likelihood method to approximate the aforementioned conditional probability.…”

Section: Introductionmentioning

confidence: 99%

TEA: A Sequential Recommendation Framework via Temporally Evolving Aggregations

Li¹,

Cai²,

Wu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Sequential recommendation aims to choose the most suitable items for a user at a specific timestamp given historical behaviors. Existing methods usually model the user behavior sequence based on the transition-based methods like Markov Chain. However, these methods also implicitly assume that the users are independent of each other without considering the influence between users. In fact, this influence plays an important role in sequence recommendation since the behavior of a user is easily affected by others. Therefore, it is desirable to aggregate both user behaviors and the influence between users, which are evolved temporally and involved in the heterogeneous graph of users and items. In this paper, we incorporate dynamic user-item heterogeneous graphs to propose a novel sequential recommendation framework. As a result, the historical behaviors as well as the influence between users can be taken into consideration. To achieve this, we firstly formalize sequential recommendation as a problem to estimate conditional probability given temporal dynamic heterogeneous graphs and user behavior sequences. After that, we exploit the conditional random field to aggregate the heterogeneous graphs and user behaviors for probability estimation, and employ the pseudo-likelihood approach to derive a tractable objective function. Finally, we provide scalable and flexible implementations of the proposed framework. Experimental results on three real-world datasets not only demonstrate the effectiveness of our proposed method but also provide some insightful discoveries on sequential recommendation.

show abstract

Semi-supervised disentangled framework for transferable named entity recognition

Cited by 14 publications

References 9 publications

Disentangled Graph Recurrent Network for Document Ranking

Disentangled Graph Recurrent Network for Document Ranking

FragNet, a Contrastive Learning-Based Transformer Model for Clustering, Interpreting, Visualizing, and Navigating Chemical Space

TEA: A Sequential Recommendation Framework via Temporally Evolving Aggregations

Contact Info

Product

Resources

About