A Qualitative Evaluation Framework for Paraphrase Identification

Kovatchev, Venelin; Martí, M. Antònia; Salamó, Maria; Beltrán, Javier

doi:10.26615/978-954-452-056-4_067

Cited by 8 publications

(6 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The task of identifying paraphrases consists in determining whether two sentences have the same meaning, and can be casted-at least from a definitional perspective-as recognizing bidirectional entailments. Pruthi et al (2019) show that computational models underperform in MRPC (Dolan et al, 2004) with adversarial misspellings, and Kovatchev et al (2019) present a qualitative analysis of 11 state-of-the-art models (overall accuracies: 68-84%). When negation is present, however, accuracies drop to 33% (6 models) 67% (4 models) and 1% (1 model).…”

Section: Previous Workmentioning

confidence: 96%

An Analysis of Natural Language Inference Benchmarks through the Lens of Negation

Hossain

Kovatchev

Dutta³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

Negation is underrepresented in existing natural language inference benchmarks. Additionally, one can often ignore the few negations in existing benchmarks and still make the right inference judgments. In this paper, we present a new benchmark for natural language inference in which negation plays an important role. We also show that state-of-the-art transformers struggle making inference judgments with the new pairs.

show abstract

Section: Previous Workmentioning

confidence: 96%

An Analysis of Natural Language Inference Benchmarks through the Lens of Negation

Hossain

Kovatchev

Dutta³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In contrast, deep learning neural networks, such as bidirectional long short-term memory (BiLSTM; Graves & Schmidhuber, 2005) and Transformer neural networks such as Bidirectional Encoder Representation from Transformers or “BERT” and “distilled” “DistilBERT” (Sanh et al, 2020), do not require feature engineering. Deep learning neural networks exhibit better performance than traditional machine learning algorithms in a variety of language processing tasks such as paraphrase identification (Kovatchev et al, 2019). We tested whether machine learning and deep learning automated scoring systems can provide an end-to-end solution for scoring ToM whereby the system takes children’s open-ended responses to the Silent Film and Strange Stories tasks and returns assigned scores for each item, replacing the need for manual scoring.…”

Section: Measuring “Advanced” Tommentioning

confidence: 99%

Machine learning and deep learning systems for automated measurement of “advanced” theory of mind: Reliability and validity in children and adolescents.

Devine¹,

Kovatchev²,

Traynor³

et al. 2023

Psychological Assessment

Self Cite

View full text Add to dashboard Cite

Understanding individual differences in theory of mind (ToM; the ability to attribute mental states to others) in middle childhood and adolescence hinges on the availability of robust and scalable measures. Open-ended response tasks yield valid indicators of ToM but are labor intensive and difficult to compare across studies. We examined the reliability and validity of new machine learning and deep learning neural network automated scoring systems for measuring ToM in children and adolescents. Two large samples of British children and adolescents aged between 7 and 13 years (Sample 1: N = 1,135, Mage = 10.22 years, SD = 1.45; Sample 2: N = 1,020, Mage = 10.36 years, SD = 1.27) completed the silent film and strange stories tasks. Teachers rated Sample 2 children’s social competence with peers. A single latent-factor explained variation in performance on both the silent film and strange stories task (in Sample 1 and 2) and test performance was sensitive to age-related differences and individual differences within each age-group. A deep learning neural network automated scoring system trained on Sample 1 exhibited interrater reliability and measurement invariance with manual ratings in Sample 2. Validity of ratings from the automated scoring system was supported by unique positive associations between ToM and teacher-rated social competence. The results demonstrate that reliable and valid measures of ToM can be obtained using the new freely available deep learning neural network automated scoring system to rate open-ended text responses.

show abstract

“…p. ex. : Androutsopoulos & Malakasiotis, 2010 ;Klemen & Robnik-Šikonja, 2021 ;Kovatchev et al, 2020 ;Marsi et al, 2007 ;Martin, 1976 ;Mel'čuk, 1988Mel'čuk, , 1992Mel'čuk & Milićević, 2014 ;Milićević, 2007 ;Zhou et al, 2022), cf. aussi les travaux sur la similarité sémantique des textes dont les paraphrases sont un cas spécial (p. ex.…”

Section: Inférences Et Paraphrasesunclassified

Inférences textuelles et constructions à verbes supports

Banyś¹

2022

NEO

View full text Add to dashboard Cite

Pour comprendre le langage naturel, les systèmes automatiques doivent avoir la capacité et la possibilité de savoir ce qui est inféré, et si oui, comment cela est inféré, et à partir de quoi dans un texte. Les implications linguistiquement déterminées imposées par un prédicat sur ses arguments propositionnels, apparaissant dans un texte comme des inférences textuelles, sont, avec d'autres éléments, tels que la connaissance du monde, les hypothèses des locuteurs sur l'utilisation de la langue, les stéréotypes situationnels, les implicatures, etc. l'un des éléments nécessaires pour qu'un système soit considéré comme comprenant le langage naturel. Dans cet article, nous nous concentrons sur l'une des parties importantes des inférences textuelles, qui n'a pratiquement pas été étudiée, à savoir les relations entre une catégorie particulière d'inférences textuelles constituée d'implicatifs phrastiques au sens de Karttunen (2012) et une catégorie particulière d'expressions polylexicales constituée de verbes supports. Nous donnons d'abord une brève introduction à l'état actuel de la description des deux catégories générales : les inférences, dont un type particulier d'inférences : les paraphrases (sec. 1), et les expressions polylexicales avec un type particulier de ces constructions : les verbes supports (sec. 2), ce qui nous permet de présenter les relations entre les verbes supports et les paraphrases (sec. 3). Nous passons ensuite à une discussion sur les implicatifs, y compris les implicatifs phrastiques (sec. 4), au sens de Karttunen (2012) et nous présentons une esquisse de la description de certains types de constructions à verbes supports du point de vue de leur pouvoir implicatif (sec. 5). Les analyses du type présenté sont à poursuivre et doivent être étendues à toutes les constructions du type analysé et font en même temps partie d'un projet beaucoup plus général. Elles sont un début d'une implémentation systématique à faire sur les matériaux français et polonais en corrélation avec l'anglais de la combinatoire des valeurs de vérité, "signatures implicatives", des prédicats, qu'ils soient verbaux, adjectivaux ou nominaux.

show abstract

A Qualitative Evaluation Framework for Paraphrase Identification

Cited by 8 publications

References 16 publications

An Analysis of Natural Language Inference Benchmarks through the Lens of Negation

An Analysis of Natural Language Inference Benchmarks through the Lens of Negation

Machine learning and deep learning systems for automated measurement of “advanced” theory of mind: Reliability and validity in children and adolescents.

Inférences textuelles et constructions à verbes supports

Contact Info

Product

Resources

About