BIBI System Description: Building with CNNs and Breaking with Deep Reinforcement Learning

Abstract

This paper describes our submission to the sentiment analysis sub-task of "Build It, Break It: The Language Edition (BIBI)", on both the builder and breaker sides. As a builder, we use convolutional neural nets, trained on both phrase and sentence data. As a breaker, we use Q-learning to learn minimal change pairs, and apply a token substitution method automatically. We analyse the results to gauge the robustness of NLP systems.

ACES: Translation Accuracy Challenge Sets at WMT 2023

“…Challenge sets have been used for several tasks (Li et al (2017); McCoy and Linzen (2019); Ravichander et al (2021), inter alia) to investigate the behaviour of these tasks under a specific phenomenon rather than the standard test distribution (Popović and Castilho, 2019). Lately, with the success of neural metrics, the development of challenge sets for MT evaluation has promoted great interest in studying the strengths and weaknesses of these metrics.…”

“…Challenge sets exist for a range of natural language processing (NLP) tasks including Sentiment Analysis (Li et al, 2017;Mahler et al, 2017;Staliūnaitė and Bonfil, 2017), Natural Language Inference (McCoy and Linzen, 2019;Rocchietti et al, 2021), Question Answering (Ravichander * Equal contribution by all authors. et al, 2021), Machine Reading Comprehension (Khashabi et al, 2018), Machine Translation (MT) (King and Falkedal, 1990;Isabelle et al, 2017), and the more specific task of pronoun translation in MT (Guillou and Hardmeier, 2016).…”

Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task

“…The builder team from University of Melbourne (which also participated as a breaker team), contributed two sentiment analysis systems consisting of convolutional neural networks. One CNN was trained on data labeled at the phrase level (PCNN), and the other was trained on data labeled at the sentence level (SCNN) (Li et al, 2017).…”

“…The breaker team from University of Melbourne opted to generate test minimal pairs automatically, borrowing from methods for generating adversarial examples in computer vision. They used reinforcement learning, optimizing on reversed labels, to identify tokens or phrases to be changed, and then applied a substitution method (Li et al, 2017). Some human supervision was used to ensure grammaticality and correct labeling of the sentences.…”