Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites

Tikhonov, Alexey; Shibaev, Viacheslav; Nagaev, Aleksander; Nugmanova, Aigul; Yamshchikov, Ivan P.

doi:10.18653/v1/d19-1406

Cited by 31 publications

(33 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Providing a meaningful comparison of our approach to existing style transfer systems is difficult because of (1) poorly-defined automatic and human methods for measuring style transfer quality (Pang, 2019;Mir et al, 2019;Tikhonov et al, 2019), and (2) misleading (or absent) methods of aggregating three individual metrics (transfer accuracy, semantic similarity and fluency) into a single number. In this section, we describe the flaws in existing metrics and their aggregation (the latter illustrated through a naïve baseline), and we propose a new evaluation methodology to fix these issues.…”

Section: Evaluating Style Transfermentioning

confidence: 99%

Reformulating Unsupervised Style Transfer as Paraphrase Generation

Krishna¹,

Wieting²,

Iyyer³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

155

149

View full text Add to dashboard Cite

Modern NLP defines the task of style transfer as modifying the style of a given sentence without appreciably changing its semantics, which implies that the outputs of style transfer systems should be paraphrases of their inputs. However, many existing systems purportedly designed for style transfer inherently warp the input's meaning through attribute transfer, which changes semantic properties such as sentiment. In this paper, we reformulate unsupervised style transfer as a paraphrase generation problem, and present a simple methodology based on fine-tuning pretrained language models on automatically generated paraphrase data. Despite its simplicity, our method significantly outperforms state-of-the-art style transfer systems on both human and automatic evaluations. We also survey 23 style transfer papers and discover that existing automatic metrics can be easily gamed and propose fixed variants. Finally, we pivot to a more real-world style transfer setting by collecting a large dataset of 15M sentences in 11 diverse styles, which we use for an in-depth analysis of our system.

show abstract

Section: Evaluating Style Transfermentioning

confidence: 99%

Reformulating Unsupervised Style Transfer as Paraphrase Generation

Krishna¹,

Wieting²,

Iyyer³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

155

149

View full text Add to dashboard Cite

show abstract

“…Whereas, the second category look to explicitly separate attributes from the content. This constraint is enforced using either adversarial training (Fu et al, 2017;Hu et al, 2017;Zhang et al, 2018;Yamshchikov et al, 2019) or MI minimisation using vCLUB-S (Cheng et al, 2020b). Traditional adversarial training is based on an encoder that aims to fool the adversary discriminator by removing attribute information from the content embedding (Elazar and Goldberg, 2018).…”

Section: Main Definitions and Related Workmentioning

confidence: 99%

“…To the best of our knowledge, a depth study of the relationship between disentangled representa-tions based either on adversarial losses solely or on vCLU B − S and quality of the generated sentences remains overlooked. Most of the previous studies have been focusing on either trade-offs between metrics computed on the generated sentences (Tikhonov et al, 2019) or performance evaluation of the disentanglement as part of (or convoluted with) more complex modules. This enhances the need to provide a fair evaluation of disentanglement methods by isolating their individual contributions (Yamshchikov et al, 2019;Cheng et al, 2020b).…”

Section: Introductionmentioning

confidence: 99%

“…Most of the previous studies have been focusing on either trade-offs between metrics computed on the generated sentences (Tikhonov et al, 2019) or performance evaluation of the disentanglement as part of (or convoluted with) more complex modules. This enhances the need to provide a fair evaluation of disentanglement methods by isolating their individual contributions (Yamshchikov et al, 2019;Cheng et al, 2020b). Methods to enforce disentangled representations can be grouped into two different categories.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations

Colombo¹,

Piantanida²,

Clavel³

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Learning disentangled representations of textual data is essential for many natural language tasks such as fair classification, style transfer and sentence generation, among others. The existent dominant approaches in the context of text data either rely on training an adversary (discriminator) that aims at making attribute values difficult to be inferred from the latent code or rely on minimising variational bounds of the mutual information between latent code and the value attribute. However, the available methods suffer of the impossibility to provide a fine-grained control of the degree (or force) of disentanglement. In contrast to adversarial methods, which are remarkably simple, although the adversary seems to be performing perfectly well during the training phase, after it is completed a fair amount of information about the undesired attribute still remains. This paper introduces a novel variational upper bound to the mutual information between an attribute and the latent code of an encoder. Our bound aims at controlling the approximation error via the Renyi's divergence, leading to both better disentangled representations and in particular, a precise control of the desirable degree of disentanglement than state-of-the-art methods proposed for textual data. Furthermore, it does not suffer from the degeneracy of other losses in multi-class scenarios. We show the superiority of this method on fair classification and on textual style transfer tasks. Additionally, we provide new insights illustrating various trade-offs in style transfer when attempting to learn disentangled representations and quality of the generated sentence.

show abstract

“…Most work in this growing field has focused primarily on style transfer within English, while covering different languages has received disproportional interest. Concretely, out of 35 ST papers we reviewed, all of them report results for ST within English text, while there is just a single work covering each of the following languages: Chinese, Russian, Latvian, Estonian, and French (Shang et al, 2019;Tikhonov et al, 2019;Korotkova et al, 2019;Niu et al, 2018). Notably, even though some efforts have been made towards multilingual ST, researchers are limited to providing system outputs as a means of evaluation, and progress is hampered by the scarcity of resources for most languages.…”

Section: Introductionmentioning

confidence: 99%