Posterior Differential Regularization with f-divergence for Improving Model Robustness

Abstract

We address the problem of enhancing model robustness through regularization. Specifically, we focus on methods that regularize the model posterior difference between clean and noisy inputs. Theoretically, we provide a connection of two recent methods, Jacobian Regularization and Virtual Adversarial Training, under this framework. Additionally, we generalize the posterior differential regularization to the family of f -divergences and characterize the overall framework in terms of Jacobian matrix. Empirically, we compare those regularizations and standard BERT training on a diverse set of tasks to provide a comprehensive profile of their effect on model generalization. For both fully supervised and semi-supervised settings, we show that regularizing the posterior difference with f -divergence can result in well-improved model robustness. In particular, with a proper f -divergence, a BERT-base model can achieve comparable generalization as its BERT-large counterpart for in-domain, adversarial and domain shift scenarios, indicating the great potential of the proposed framework for enhancing NLP model robustness. 1

UnitedQA: A Hybrid Approach for Open Domain Question Answering

“…In addition to better textual representations from ELECTRA (Clark et al, 2020) (Cheng et al, 2020b) to improve the robustness of the extractive reader. As an extension to recent methods for improving the model local smoothness (Miyato et al, 2018;Sokolić et al, 2017), PDR aims at regularizing the posterior difference between the clean and noisy inputs with regard to the family of f -divergences (Csiszár and Shields, 2004).…”

“…As an extension to recent methods for improving the model local smoothness (Miyato et al, 2018;Sokolić et al, 2017), PDR aims at regularizing the posterior difference between the clean and noisy inputs with regard to the family of f -divergences (Csiszár and Shields, 2004). Different from Cheng et al (2020b) where only clean supervision setting is considered, in this work, we apply PDR to the weakly supervised open-domain QA scenario. Given it is computationally expensive to enumerate all possible spans, we apply two separate regularization terms for the begin and end position probabilities at the multi-passage level, respectively.…”

“…In this section, we explore different approaches to improving the extractive model for open domain QA. Here, we mainly consider recent advances in 1) improving NLP model robustness (Cheng et al, 2020b), 2) extractive QA with weak supervision (Min et al, 2019;Cheng et al, 2020a), and 3) enhanced textual representations (Clark et al, 2020).…”

“…T5 (Raffel et al, 2019) and ELECTRA (Clark et al, 2020), in the United-QA, we further study techniques to improve and stabilize the model training for both extractive and generative readers. Specifically, we consider posterior differential regularization (Cheng et al, 2020b) and distant supervision assumptions (Cheng et al, 2020a) to enhance the extractive reader. For the generative reader, we incorporate attention bias (Lewis et al, 2020a) into T5-FID (Izacard and Grave, 2020), and improve unconstrained generation training with adversarial training (Ju et al, 2019;Jiang et al, 2020).…”

Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention

“…Previous works show that virtual adversarial training (VAT, Miyato et al ( 2018)) can improve the performance for general NLU and question answering tasks (Jiang et al, 2020;Cheng et al, 2021). In the multiple-choice commonsense reasoning task, the goal is to minimize the cross-entropy loss:…”