Paraphrase Detection on Noisy Subtitles in Six Languages

Abstract

We perform automatic paraphrase detection on subtitle data from the Opusparcus corpus comprising six European languages: German, English, Finnish, French, Russian, and Swedish. We train two types of supervised sentence embedding models: a word-averaging (WA) model and a gated recurrent averaging network (GRAN) model. We find out that GRAN outperforms WA and is more robust to noisy training data. Better results are obtained with more and noisier data than less and cleaner data. Additionally, we experiment on other datasets, without reaching the same level of performance, because of domain mismatch between training and test data.

Coping with Noisy Training Data Labels in Paraphrase Detection

“…We sample our training examples similarly to Sjöblom et al (2018), who picked a certain number of assumed positive examples of paraphrases from the beginning of the training sets, and sampled an equal number of assumed negative examples by randomly pairing sentences. Sjöblom et al (2018) experimented with data sets containing between one and thirty million positive examples. The level of correct labels in these subsets of data are shown in Table 1, which has been compiled from values reported by Creutz (2018) and Sjöblom et al (2018).…”

“…Sjöblom et al (2018) experimented with data sets containing between one and thirty million positive examples. The level of correct labels in these subsets of data are shown in Table 1, which has been compiled from values reported by Creutz (2018) and Sjöblom et al (2018). The reported best models were trained on data sets containing between 60 % and 80 % of presumably correctly labeled paraphrases.…”

“…Paraphrase detection based on Opusparcus has previously been studied by Sjöblom et al (2018), who train two different neural network models to represent the phrases and apply a cosine distancebased metric to detect the paraphrases. The authors observe that the networks are rather robust to noise and they can benefit from more data, albeit increase in data size results in increased amounts of noise within the training data.…”

“…The authors observe that the networks are rather robust to noise and they can benefit from more data, albeit increase in data size results in increased amounts of noise within the training data. Whereas Sjöblom et al (2018) use neural networks trained from scratch, we approach paraphrase detection by fine-tuning language-specific BERT models for sequence classification.…”

The Korean Morphologically Tight-Fitting Tokenizer for Noisy User-Generated Texts

“…Michel and Neubig (2018) provided types of noises observed in social media texts and focused on spelling and grammatical errors, emojis, and profanities to perform machine translation. Sjöblom et al (2018) defined noises for subtitles including misspellings, misalignments, and sentence segmentation errors. Agarwal et al (2020) summarized noise types in free-text answers written by Hindi children, such as punctuations, emojis, translated/transliterated text, missing space between words, and so on.…”