Annotation Guidelines for the Turku Paraphrase Corpus

Kanerva, Jenna; Ginter, Filip; Chang, Lynn; Rastas, Iiro; Skantsi, Valtteri; Kilpeläinen, Jemina; Kupari, Hanna-Mari; Piirto, Aurora; Saarni, Jenna; Sevón, Maija; Tarkka, Otto

doi:10.48550/arxiv.2108.07499

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These flags are independent of each other and thus one label 4 paraphrase pair can have multiple flags, disregarding the directional subsumption flags. More detailed description of the labels together with example annotations is given in the annotation guidelines (Kanerva et al 2021a).…”

Section: Methodsmentioning

confidence: 99%

Textual Paraphrase Dataset for Deep Language Modelling

Kanerva

Ginter

Chang

et al. 2022

European Language Grid

View full text Add to dashboard Cite

The Turku Paraphrase Corpus is a dataset of over 100,000 Finnish paraphrase pairs. During the corpus creation, we strived to gather challenging paraphrase pairs, more suitable to test the capabilities of natural language understanding models. The paraphrases are both selected and classified manually, so as to minimise lexical overlap, and provide examples that are structurally and lexically different to the maximum extent. An important distinguishing feature of the corpus is that most of the paraphrase pairs are extracted and distributed in their native document context, rather than in isolation. The primary application for the dataset is the development and evaluation of deep language models, and representation learning in general.

show abstract

Section: Methodsmentioning

confidence: 99%

Textual Paraphrase Dataset for Deep Language Modelling

Kanerva

Ginter

Chang

et al. 2022

European Language Grid

View full text Add to dashboard Cite

show abstract

“…A total of 17page annotation manual was produced in collaboration among the annotators, and the guidelines were revised and extended regularly to account for new problematic cases. The full manual is published as a technical report (Kanerva et al 2021a), and some of the most interesting/relevant policies are discussed below.…”

Section: Annotation Guidelinesmentioning

confidence: 99%

Towards diverse and contextually anchored paraphrase modeling: A dataset and baselines for Finnish

et al. 2023

Self Cite

View full text Add to dashboard Cite

In this paper, we study natural language paraphrasing from both corpus creation and modeling points of view. We focus in particular on the methodology that allows the extraction of challenging examples of paraphrase pairs in their natural textual context, leading to a dataset potentially more suitable for evaluating the models’ ability to represent meaning, especially in document context, when compared with those gathered using various sentence-level heuristics. To this end, we introduce the Turku Paraphrase Corpus, the first large-scale, fully manually annotated corpus of paraphrases in Finnish. The corpus contains 104,645 manually labeled paraphrase pairs, of which 98% are verified to be true paraphrases, either universally or within their present context. In order to control the diversity of the paraphrase pairs and avoid certain biases easily introduced in automatic candidate extraction, the paraphrases are manually collected from different paraphrase-rich text sources. This allows us to create a challenging dataset including longer and more lexically diverse paraphrases than can be expected from those collected through heuristics. In addition to quality, this also allows us to keep the original document context for each pair, making it possible to study paraphrasing in context. To our knowledge, this is the first paraphrase corpus which provides the original document context for the annotated pairs. We also study several paraphrase models trained and evaluated on the new data. Our initial paraphrase classification experiments indicate a challenging nature of the dataset when classifying using the detailed labeling scheme used in the corpus annotation, the accuracy substantially lacking behind human performance. However, when evaluating the models on a large scale paraphrase retrieval task on almost 400M candidate sentences, the results are highly encouraging, 29–53% of the pairs being ranked in the top 10 depending on the paraphrase type. The Turku Paraphrase Corpus is available at github.com/TurkuNLP/Turku-paraphrase-corpus as well as through the popular HuggingFace datasets under the CC-BY-SA license.

show abstract

Annotation Guidelines for the Turku Paraphrase Corpus

Cited by 2 publications

References 0 publications

Textual Paraphrase Dataset for Deep Language Modelling

Textual Paraphrase Dataset for Deep Language Modelling

Towards diverse and contextually anchored paraphrase modeling: A dataset and baselines for Finnish

Contact Info

Product

Resources

About