When data permutations are pathological: the case of neural natural language inference

Schluter, Natalie; Varab, Daniel

doi:10.18653/v1/d18-1534

Cited by 10 publications

(9 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These techniques were until recently quite rare in this field, despite the inherently repeatable nature of most natural language processing experiments. Researchers attempting replications or reproductions have reported problems with availability of data (Mieskes, 2017;Wieling et al, 2018) and software (Pedersen, 2008), and various details of implementation (Fokkens et al, 2013;Reimers and Gurevych, 2017;Schluter and Varab, 2018). While we cannot completely avoid these pitfalls, we select a task-English part-ofspeech tagging-for which both data and software are abundantly available.…”

Section: Replication and Reproductionmentioning

confidence: 99%

We Need to Talk about Standard Splits

Gorman¹,

Bedrick²

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

It is standard practice in speech & language technology to rank systems according to performance on a test set held out for evaluation. However, few researchers apply statistical tests to determine whether differences in performance are likely to arise by chance, and few examine the stability of system ranking across multiple training-testing splits. We conduct replication and reproduction experiments with nine part-of-speech taggers published between 2000 and 2018, each of which reports state-of-the-art performance on a widely-used "standard split". We fail to reliably reproduce some rankings using randomly generated splits. We suggest that randomly generated splits should be used in system comparison.

show abstract

Section: Replication and Reproductionmentioning

confidence: 99%

We Need to Talk about Standard Splits

Gorman¹,

Bedrick²

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…However, they are known for being "black boxes" which are not easily interpretable. Recent interest in interpreting these methods has led to new lines of research which attempt to discover what linguistic phenomena neural networks are able to learn (Linzen et al, 2016;Gulordava et al, 2018;Conneau et al, 2018), how robust neural networks are to perturbations in input data (Ribeiro et al, 2018;Ebrahimi et al, 2018;Schluter and Varab, 2018), and what biases they propagate (Park et al, 2018;Zhao et al, 2018;Kiritchenko and Mohammad, 2018).…”

Section: Introductionmentioning

confidence: 99%

Sentiment Analysis Is Not Solved! Assessing and Probing Sentiment Classification

Barnes

Øvrelid

Velldal

2019

Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

Neural methods for SA have led to quantitative improvements over previous approaches, but these advances are not always accompanied with a thorough analysis of the qualitative differences. Therefore, it is not clear what outstanding conceptual challenges for sentiment analysis remain. In this work, we attempt to discover what challenges still prove a problem for sentiment classifiers for English and to provide a challenging dataset. We collect the subset of sentences that an (oracle) ensemble of state-of-the-art sentiment classifiers misclassify and then annotate them for 18 linguistic and paralinguistic phenomena, such as negation, sarcasm, modality, etc. 1 Finally, we provide a case study that demonstrates the usefulness of the dataset to probe the performance of a given sentiment classifier with respect to linguistic phenomena.

show abstract

“…The SNLI data set is a text implication recognition data set published by Stanford University. SNLI is manually annotated and contains 570 k text pairs, used as testing and training sets for NLI systems [25][26][27][28]. There are three kinds of marks: implication, contradiction, and neutral.…”

Section: Snli Datasetmentioning

confidence: 99%

Sentence Representation Method Based on Multi-Layer Semantic Network

2021

View full text Add to dashboard Cite

With the development of artificial intelligence, more and more people hope that computers can understand human language through natural language technology, learn to think like human beings, and finally replace human beings to complete the highly difficult tasks with cognitive ability. As the key technology of natural language understanding, sentence representation reasoning technology mainly focuses on the sentence representation method and the reasoning model. Although the performance has been improved, there are still some problems such as incomplete sentence semantic expression, lack of depth of reasoning model, and lack of interpretability of the reasoning process. In this paper, a multi-layer semantic representation network is designed for sentence representation. The multi-attention mechanism obtains the semantic information of different levels of a sentence. The word order information of the sentence is also integrated by adding the relative position mask between words to reduce the uncertainty caused by word order. Finally, the method is verified on the task of text implication recognition and emotion classification. The experimental results show that the multi-layer semantic representation network can promote sentence representation’s accuracy and comprehensiveness.

show abstract

When data permutations are pathological: the case of neural natural language inference

Cited by 10 publications

References 6 publications

We Need to Talk about Standard Splits

We Need to Talk about Standard Splits

Sentiment Analysis Is Not Solved! Assessing and Probing Sentiment Classification

Sentence Representation Method Based on Multi-Layer Semantic Network

Contact Info

Product

Resources

About