Textual Paraphrase Dataset for Deep Language Modelling

Kanerva, Jenna; Ginter, Filip; Chang, Li-Hsin; Skantsi, Valtteri; Kilpeläinen, Jemina; Kupari, Hanna-Mari; Piirto, Aurora; Saarni, Jenna; Sevón, Maija; Tarkka, Otto

doi:10.1007/978-3-031-17258-8_27

Cited by 4 publications

(5 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Turku Paraphrase Corpus introduced in this paper, the first large-scale, manually annotated paraphrase corpus for Finnish, includes 91,604 manually extracted and labeled paraphrases with an additional 13,041 human-made rephrasing of statements. While the first incomplete version of the corpus was released in Kanerva et al (2021b), the current work extends the contributions into multiple directions: (1) the corpus size is doubled from the first release, (2) the text sources used to gather the paraphrases are extended from alternative subtitles and news headings to include also news articles, university student essays, translation exercises made by university students, as well as messages from an online discussion forum, (3) each manually extracted paraphrase is distributed together with the original document context to allow studies on paraphrasing in context, (4) in addition to manually extracted and labeled paraphrases, an automatically extracted subset of the corpus that contains related nonparaphrase segments is provided to support paraphrase classification.…”

Section: Resources For Finnishmentioning

confidence: 99%

“…Our paraphrase classification model is a pairwise classifier based on the BERT encoder, following our initial work reported in Kanerva et al (2021b)…”

Section: Paraphrase Classifiermentioning

confidence: 99%

Section: Resources For Finnishmentioning

confidence: 99%

See 2 more Smart Citations

Towards diverse and contextually anchored paraphrase modeling: A dataset and baselines for Finnish

et al. 2023

Self Cite

View full text Add to dashboard Cite

In this paper, we study natural language paraphrasing from both corpus creation and modeling points of view. We focus in particular on the methodology that allows the extraction of challenging examples of paraphrase pairs in their natural textual context, leading to a dataset potentially more suitable for evaluating the models’ ability to represent meaning, especially in document context, when compared with those gathered using various sentence-level heuristics. To this end, we introduce the Turku Paraphrase Corpus, the first large-scale, fully manually annotated corpus of paraphrases in Finnish. The corpus contains 104,645 manually labeled paraphrase pairs, of which 98% are verified to be true paraphrases, either universally or within their present context. In order to control the diversity of the paraphrase pairs and avoid certain biases easily introduced in automatic candidate extraction, the paraphrases are manually collected from different paraphrase-rich text sources. This allows us to create a challenging dataset including longer and more lexically diverse paraphrases than can be expected from those collected through heuristics. In addition to quality, this also allows us to keep the original document context for each pair, making it possible to study paraphrasing in context. To our knowledge, this is the first paraphrase corpus which provides the original document context for the annotated pairs. We also study several paraphrase models trained and evaluated on the new data. Our initial paraphrase classification experiments indicate a challenging nature of the dataset when classifying using the detailed labeling scheme used in the corpus annotation, the accuracy substantially lacking behind human performance. However, when evaluating the models on a large scale paraphrase retrieval task on almost 400M candidate sentences, the results are highly encouraging, 29–53% of the pairs being ranked in the top 10 depending on the paraphrase type. The Turku Paraphrase Corpus is available at github.com/TurkuNLP/Turku-paraphrase-corpus as well as through the popular HuggingFace datasets under the CC-BY-SA license.

show abstract

Section: Resources For Finnishmentioning

confidence: 99%

“…Our paraphrase classification model is a pairwise classifier based on the BERT encoder, following our initial work reported in Kanerva et al (2021b)…”

Section: Paraphrase Classifiermentioning

confidence: 99%

Section: Resources For Finnishmentioning

confidence: 99%

See 1 more Smart Citation

Towards diverse and contextually anchored paraphrase modeling: A dataset and baselines for Finnish

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the generative task, we use a paraphrase dataset from (Kanerva et al, 2021). We use 21k for training, 2.6k for validation, and 2.6k for testing.…”

Section: Downstream Tasksmentioning

confidence: 99%

Effects of sub-word segmentation on performance of transformer language models

Hou,

Katinskaia,

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Language modeling is a fundamental task in natural language processing, which has been thoroughly explored with various architectures and hyperparameters. However, few studies focus on the effect of sub-word segmentation on the performance of language models (LMs). In this paper, we compare GPT and BERT models trained with the statistical segmentation algorithm BPE vs. two unsupervised algorithms for morphological segmentation-Morfessor and StateMorph. We train the models for several languages-including ones with very rich morphology-and compare their performance with different segmentation algorithms, vocabulary sizes, and model sizes. The results show that training with morphological segmentation allows the LMs to: 1. achieve lower perplexity, 2. converge more efficiently in terms of training time, and 3. achieve equivalent or better evaluation scores on downstream tasks. Lastly, we show 4. that LMs of smaller size using morphological segmentation can perform comparably to models of larger size trained with BPEboth in terms of (1) perplexity and (3) scores on downstream tasks. Points 2 and 4 impact on sustainability, since they reduce the model cost; and while 2 reduces cost only in the training phase, 4 does so also in the inference phase.

show abstract

“…Finnish and Swedish The Finnish Paraphrase Corpus (Kanerva et al, 2021), a dataset of manually selected subtitle lines and news headlines, annotated on a four-class ordinal scale. A small test set in Swedish is likewise available.…”

Section: Datasetsmentioning

confidence: 99%

NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measures

Vamvas¹,

Sennrich²

2022

Findings of the Association for Computational Linguistics: EMNLP 2022

View full text Add to dashboard Cite

Being able to rank the similarity of short text segments is an interesting bonus feature of neural machine translation. Translation-based similarity measures include direct and pivot translation probability, as well as translation cross-likelihood, which has not been studied so far. We analyze these measures in the common framework of multilingual NMT, releasing the NMTSCORE library. Compared to baselines such as sentence embeddings, translation-based measures prove competitive in paraphrase identification and are more robust against adversarial or multilingual input, especially if proper normalization is applied. When used for reference-based evaluation of data-to-text generation in 2 tasks and 17 languages, translation-based measures show a relatively high correlation to human judgments.

show abstract

Textual Paraphrase Dataset for Deep Language Modelling

Cited by 4 publications

References 2 publications

Towards diverse and contextually anchored paraphrase modeling: A dataset and baselines for Finnish

Towards diverse and contextually anchored paraphrase modeling: A dataset and baselines for Finnish

Effects of sub-word segmentation on performance of transformer language models

NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measures

Contact Info

Product

Resources

About