Enhancing Interpretable Clauses Semantically using Pretrained Word Representation

Yadav, Rohan Kumar; Jiao, Lei; Granmo, Ole‐Christoffer; Goodwin, Morten

doi:10.18653/v1/2021.blackboxnlp-1.19

Cited by 9 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Barhom et al, 2019;Yadav et al, 2021a). For a fair comparison, we report the baseline performance by re-running Cattan et al (2021a) using gold mentions similar to the baseline used in Yadav et al (2021b). We compare this baseline to two variants of our model, based on intra-span and inter-span attention (Sec 3.2).…”

Section: Implementation Detailsmentioning

confidence: 99%

What happens before and after: Multi-Event Commonsense in Event Coreference Resolution

Ravi¹,

Tanner²,

Ng³

et al. 2023

Preprint

View full text Add to dashboard Cite

Event coreference models cluster event mentions pertaining to the same real-world event.Recent models rely on contextualized representations to recognize coreference among lexically or contextually similar mentions. However, models typically fail to leverage commonsense inferences, which is particularly limiting for resolving lexically-divergent mentions. We propose a model that extends event mentions with temporal commonsense inferences. Given a complex sentence with multiple events, e.g., "The man killed his wife and got arrested", with the target event "arrested", our model generates plausible events that happen before the target event -such as "the police arrived", and after it, such as "he was sentenced". We show that incorporating such inferences into an existing event coreference model improves its performance, and we analyze the coreferences in which such temporal knowledge is required.

show abstract

Section: Implementation Detailsmentioning

confidence: 99%

What happens before and after: Multi-Event Commonsense in Event Coreference Resolution

Ravi¹,

Tanner²,

Ng³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The performance is slightly below Bi-LSTM/GRU-based model because of its restriction to use pre-trained word embedding. However, Yadav et al [22] show that embedding similar words using a pre-trained word embedding significantly enhances the performance and outperforms the baselines. However, our proposed model only uses TM explainability to generate prerequisite word weightage to replace human attention input into neural network language models.…”

Section: Performance Comparison With State-of-the-artsmentioning

confidence: 99%

“…Unlike Deep Neural Netowrks (DNNs) and basic rule-based systems, TM learns rules in the same way that humans do, using logical reasoning, and it does so in a visible and interpretable manner [16,20]. TM has shown that it obtains a good trade-off between accuracy and interpretability on many NLP tasks [21,22]. However, there lie some limitations such as boolean bag-of-words input and incapable of using pre-trained information.…”

Section: Introductionmentioning

confidence: 99%

Enhancing Attention’s Explanation Using Interpretable Tsetlin Machine

Yadav

Constantin²

2022

Algorithms

View full text Add to dashboard Cite

Explainability is one of the key factors in Natural Language Processing (NLP) specially for legal documents, medical diagnosis, and clinical text. Attention mechanism has been a popular choice for such explainability recently by estimating the relative importance of input units. Recent research has revealed, however, that such processes tend to misidentify irrelevant input units when explaining them. This is due to the fact that language representation layers are initialized by pre-trained word embedding that is not context-dependent. Such a lack of context-dependent knowledge in the initial layer makes it difficult for the model to concentrate on the important aspects of input. Usually, this does not impact the performance of the model, but the explainability differs from human understanding. Hence, in this paper, we propose an ensemble method to use logic-based information from the Tsetlin Machine to embed it into the initial representation layer in the neural network to enhance the model in terms of explainability. We obtain the global clause score for each word in the vocabulary and feed it into the neural network layer as context-dependent information. Our experiments show that the ensemble method enhances the explainability of the attention layer without sacrificing any performance of the model and even outperforming in some datasets.

show abstract

Word-level human interpretable scoring mechanism for novel text detection using Tsetlin Machines

2022

View full text Add to dashboard Cite

Recent research in novelty detection focuses mainly on document-level classification, employing deep neural networks (DNN). However, the black-box nature of DNNs makes it difficult to extract an exact explanation of why a document is considered novel. In addition, dealing with novelty at the word level is crucial to provide a more fine-grained analysis than what is available at the document level. In this work, we propose a Tsetlin Machine (TM)-based architecture for scoring individual words according to their contribution to novelty. Our approach encodes a description of the novel documents using the linguistic patterns captured by TM clauses. We then adapt this description to measure how much a word contributes to making documents novel. Our experimental results demonstrate how our approach breaks down novelty into interpretable phrases, successfully measuring novelty.

show abstract

Enhancing Interpretable Clauses Semantically using Pretrained Word Representation

Cited by 9 publications

References 22 publications

What happens before and after: Multi-Event Commonsense in Event Coreference Resolution

What happens before and after: Multi-Event Commonsense in Event Coreference Resolution

Enhancing Attention’s Explanation Using Interpretable Tsetlin Machine

Word-level human interpretable scoring mechanism for novel text detection using Tsetlin Machines

Contact Info

Product

Resources

About