Annotating and analyzing the interactions between meaning relations

Gold, Darina; Kovatchev, Venelin; Zesch, Torsten

doi:10.18653/v1/w19-4004

Cited by 8 publications

(11 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…posed by Gold et al (2019). The authors first semiautomatically generated a large pool of premisehypothesis pairs.…”

Section: Crowd "Annotation" Strategymentioning

confidence: 99%

“…The "true-false" strategy combines one true and one false sentence derived from the same premise. Unlike Gold et al (2019), we do not include a random pairing and do not downsample "false-false" and "true-false" strategies. We randomly selected 2,000 of the pairs for annotation, ensuring equal distribution of strategies.…”

Section: Pair Generationmentioning

confidence: 99%

“…We used the WARP-Text (Kovatchev et al, 2018b) annotation interface for the annotation. Following Gold et al (2019), we created two separate binary annotation tasks -one for entailment and one for contradiction. For entailment, we included each pair twice, changing the order of P and H to reflect the directional nature of the relation.…”

Section: Pair Generationmentioning

confidence: 99%

See 2 more Smart Citations

InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples

Kovatchev¹,

Taulé²

2022

Preprint

View full text Add to dashboard Cite

In this paper, we present INFERES -an original corpus for Natural Language Inference (NLI) in European Spanish. We propose, implement, and analyze a variety of corpuscreating strategies utilizing expert linguists and crowd workers. The objectives behind IN-FERES are to provide high-quality data, and, at the same time to facilitate the systematic evaluation of automated systems. Specifically, we focus on measuring and improving the performance of machine learning systems on negation-based adversarial examples and their ability to generalize across out-of-distribution topics.We train two transformer models on IN-FERES (8,055 gold examples) in a variety of scenarios. Our best model obtains 72.8% accuracy, leaving a lot of room for improvement. The "hypothesis-only" baseline performs only 2%-5% higher than majority, indicating much fewer annotation artifacts than prior work. We find that models trained on INFERES generalize very well across topics (both in-and outof-distribution) and perform moderately well on negation-based adversarial examples.

show abstract

“…posed by Gold et al (2019). The authors first semiautomatically generated a large pool of premisehypothesis pairs.…”

Section: Crowd "Annotation" Strategymentioning

confidence: 99%

Section: Pair Generationmentioning

confidence: 99%

See 1 more Smart Citation

InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples

Kovatchev¹,

Taulé²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Similar observations have been made for Textual Entailment (Sammons et al, 2010;Cabrio and Magnini, 2014). Gold et al (2019) study the interactions between paraphrasing and entailment.…”

Section: Related Workmentioning

confidence: 99%

A Qualitative Evaluation Framework for Paraphrase Identification

Kovatchev¹,

Martí²,

Salamó³

et al. 2019

Proceedings - Natural Language Processing in a Deep Learning World

Self Cite

View full text Add to dashboard Cite

In this paper, we present a new approach for the evaluation, error analysis, and interpretation of supervised and unsupervised Paraphrase Identification (PI) systems. Our evaluation framework makes use of a PI corpus annotated with linguistic phenomena to provide a better understanding and interpretation of the performance of various PI systems. Our approach allows for a qualitative evaluation and comparison of the PI models using human interpretable categories. It does not require modification of the training objective of the systems and does not place additional burden on the developers. We replicate several popular supervised and unsupervised PI systems. Using our evaluation framework we show that: 1) Each system performs differently with respect to a set of linguistic phenomena and makes qualitatively different kinds of errors; 2) Some linguistic phenomena are more challenging than others across all systems.

show abstract

“…Random sampling of examples has been successfully used in tasks such as Semantic Textual Similarity (STS) and Natural Language Inference (NLI)(Agirre et al, 2013;Gold et al, 2019) to obtain a reasonable lower-bound. Comparison with random baseline demonstrates that our system selects examples that can improve human performance.…”

mentioning

confidence: 99%

ProtoTEx: Explaining Model Decisions with Prototype Tensors

Das¹,

Chitrank²,

Kovatchev³

et al. 2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

View full text Add to dashboard Cite

We present PROTOTEX, a novel white-box NLP classification architecture based on prototype networks (Li et al., 2018). PROTOTEX faithfully explains model decisions based on prototype tensors that encode latent clusters of training examples. At inference time, classification decisions are based on the distances between the input text and the prototype tensors, explained via the training examples most similar to the most influential prototypes. We also describe a novel interleaved training algorithm that effectively handles classes characterized by the absence of indicative features. On a propaganda detection task, PROTOTEX accuracy matches BART-large and exceeds BERTlarge with the added benefit of providing faithful explanations. A user study also shows that prototype-based explanations help non-experts to better recognize propaganda in online news.

show abstract

Annotating and analyzing the interactions between meaning relations

Cited by 8 publications

References 21 publications

InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples

InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples

A Qualitative Evaluation Framework for Paraphrase Identification

ProtoTEx: Explaining Model Decisions with Prototype Tensors

Contact Info

Product

Resources

About