Simple Fusion: Return of the Language Model

Stahlberg, Felix; Cross, James; Stoyanov, Veselin

doi:10.18653/v1/w18-6321

Cited by 54 publications

(60 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Application of two countermeasures, GROVER and GLTR, to the detection of fake reviews demonstrated detection accuracy of around 90%. We plan to investigate ways to further preserve both sentiment and context information by using cold fusion [33] or simple fusion [34]. Since the generated reviews is the most probable sequence, they lack diversity and the corresponding distribution area may be already covered by the countermeasures.…”

Section: Discussionmentioning

confidence: 99%

Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-Based Detection

Adelani¹,

Mai

Fang

et al. 2020

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

Advanced neural language models (NLMs) are widely used in sequence generation tasks because they are able to produce fluent and meaningful sentences. They can also be used to generate fake reviews, which can then be used to attack online review systems and influence the buying decisions of online shoppers. A problem in fake review generation is how to generate the desired sentiment/topic. Existing solutions first generate an initial review based on some keywords and then modify some of the words in the initial review so that the review has the desired sentiment/topic. We overcome this problem by using the GPT-2 NLM to generate a large number of high-quality reviews based on a review with the desired sentiment and then using a BERT based text classifier (with accuracy of 96%) to filter out reviews with undesired sentiments. Because none of the words in the review are modified, fluent samples like the training data can be generated from the learned distribution. A subjective evaluation with 80 participants demonstrated that this simple method can produce reviews that are as fluent as those written by people. It also showed that the participants tended to distinguish fake reviews randomly. Two countermeasures, GROVER and GLTR, were found to be able to accurately detect fake review.

show abstract

Section: Discussionmentioning

confidence: 99%

Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-Based Detection

Adelani¹,

Mai

Fang

et al. 2020

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

show abstract

“…In contrast to parallel data, large quantity of monolingual data can be easily collected for many languages. Previous work proposed several ways to take advantage of monolingual data in order to improve translation models trained on parallel data, such as a separately trained language model to be integrated to the NMT system architecture (Gulcehre et al 2017;Stahlberg et al 2018) or exploiting monolingual data to create synthetic parallel data. The latter was first proposed for SMT (Schwenk 2008;Lambert et al 2011), but remains the most prevalent one in NMT (Barrault et al 2019) due to its simplicity and effectiveness.…”

Section: Low-resource Mtmentioning

confidence: 99%

Extremely low-resource neural machine translation for Asian languages

Rubino¹,

Marie²,

Dabre³

et al. 2020

Machine Translation

View full text Add to dashboard Cite

This paper presents a set of effective approaches to handle extremely low-resource language pairs for self-attention based neural machine translation (NMT) focusing on English and four Asian languages. Starting from an initial set of parallel sentences used to train bilingual baseline models, we introduce additional monolingual corpora and data processing techniques to improve translation quality. We describe a series of best practices and empirically validate the methods through an evaluation conducted on eight translation directions, based on state-of-the-art NMT approaches such as hyper-parameter search, data augmentation with forward and backward translation in combination with tags and noise, as well as joint multilingual training. Experiments show that the commonly used default architecture of self-attention NMT models does not reach the best results, validating previous work on the importance of hyper-parameter tuning. Additionally, empirical results indicate the amount of synthetic data required to efficiently increase the parameters of the models leading to the best translation quality measured by automatic metrics. We show that the best NMT models trained on large amount of tagged back-translations outperform three other synthetic data generation approaches. Finally, comparison with statistical machine translation (SMT) indicates that extremely low-resource NMT requires a large amount of synthetic parallel data obtained with back-translation in order to close the performance gap with the preceding SMT approach.

show abstract

“…Reference [5] draw an empirical roadmap to observe how the amounts of BT data impact the performance of the final system, they further investigate more factors of BT data in different SMT and NMT approaches, as well as the amounts of data [6]. Although back translation is proved effective, some works like [7] have demonstrated that back translation increasing the amount of monolingual data improves the translation quality only up to some point, and then it starts to degrade.…”

Section: B Back Translationmentioning

confidence: 99%

Revisiting Back-Translation for Low-Resource Machine Translation Between Chinese and Vietnamese

Sha

Shi

2020

IEEE Access

View full text Add to dashboard Cite

Back-translation (BT) has been widely used and become one of standard techniques for data augmentation in Neural Machine Translation (NMT), BT has proven to be beneficial for improving the performance of translation effectively, especially for low-resource scenarios. While most previous works related to BT mainly focus on European languages with high relatedness, few of them study less-related languages in other areas around the world. In this paper, we choose the language pair with less relatedness in Asia: Chinese and Vietnamese, to investigate the impacts of BT on extremely low-resource machine translation between them. We first discuss the similarities and differences between the two languages, then evaluate and compare the effects of different sizes of back-translated data on NMT and Statistical Machine Translation (SMT) models for Chinese-Vietnamese and Vietnamese-Chinese, with both characterbased and word-based settings, and conduct further analysis on the translation outputs from several aspects. Some conclusions from previous works are partially confirmed and we also draw some new findings and conclusions, which are beneficial to understand BT further and deeper for translation between less-related low-resource languages.

show abstract

Simple Fusion: Return of the Language Model

Cited by 54 publications

References 25 publications

Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-Based Detection

Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-Based Detection

Extremely low-resource neural machine translation for Asian languages

Revisiting Back-Translation for Low-Resource Machine Translation Between Chinese and Vietnamese

Contact Info

Product

Resources

About