Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization

Abstract

Text style transfer aims to alter the style (e.g., sentiment) of a sentence while preserving its content. A common approach is to map a given sentence to content representation that is free of style, and the content representation is fed to a decoder with a target style. Previous methods in filtering style completely remove tokens with style at the token level, which incurs the loss of content information. In this paper, we propose to enhance content preservation by implicitly removing the style information of each token with reverse attention, and thereby retain the content. Furthermore, we fuse content information when building the target style representation, making it dynamic with respect to the content. Our method creates not only styleindependent content representation, but also content-dependent style representation in transferring style. Empirical results show that our method outperforms the state-of-the-art baselines by a large margin in terms of content preservation. In addition, it is also competitive in terms of style transfer accuracy and fluency.

Non-Parallel Text Style Transfer with Self-Parallel Supervision

“…When comparing LaMer with the three LM-based models, we find the advantage of LaMer mainly comes from the gain in content preservation (BLEU), which demonstrates the effectiveness of our scene graph alignment. We also find some of the other TST models produce low-fluency (i.e., high perplexity) transferred sentences (see Table 10 in §A.7), though we must note that recent studies have shown perplexity to be a poor measure of fluency Mir et al, 2019;Lee et al, 2021).…”

“…As can be seen, certain methods, such as CAE and Style Transformer, have very high perplexity in more challenging TST tasks (e.g., Political Stance), potentially because the task requires longer generation and more entities involved in the sentences (see Table 7 for further discussions). However, it should also be noted that prior work have shown perplexity to be a poor measure of fluency Mir et al, 2019;Lee et al, 2021).…”

“…Text style transfer (TST) models learn how to transfer the style of text from source to target while preserving the style-independent content Fu et al, 2018). Existing TST methods perform well when transferring simple styles, such as sentiment; however, they tend to do a poor job on more abstract and subtle styles, such as formality and political stance (Lee et al, 2021;Fu et al, 2019b).…”

Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfer

“…METEOR (Mir et al, 2019;Lyu et al, 2021;Briakou et al, 2021a)) and neural-based (e.g. BERTScore (Reid and Zhong, 2021;Lee et al, 2021;Briakou et al, 2021a)). In order to further increase the capturing of semantic information beyond the lexical level, Lai et al (2021b,a) recently also employed BLEURT (Sellam et al, 2020) and COMET (Rei et al, 2020) to evaluate their systems.…”

From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Survey

“…Claiming that the conditioning structure is essential for the performance of a style transfer model, Lai et al (2019) refrained from treating the target attribute simply as part of the initial vector fed to the decoder, and instead concatenated the style vector with the output of a Gated Recurrent Unit (Chung et al 2015) cell at each time step. Style information was implicitly obfuscated at the token level by Lee et al (2021), under the assumption that the alternative option of explicit removal of tokens would result in an information loss. They opted for an adversarial strategy, which reversed the attention scores of a style discriminator to obtain a style devoid content representation, and they applied conditional layer normalization on this representation, to adapt it to the target attribute distribution.…”