Domain Agnostic Real-Valued Specificity Prediction

Abstract

Sentence specificity quantifies the level of detail in a sentence, characterizing the organization of information in discourse. While this information is useful for many downstream applications, specificity prediction systems predict very coarse labels (binary or ternary) and are trained on and tailored toward specific domains (e.g., news). The goal of this work is to generalize specificity prediction to domains where no labeled data is available and output more nuanced realvalued specificity ratings. We present an unsupervised domain adaptation system for sentence specificity prediction, specifically designed to output real-valued estimates from binary training labels. To calibrate the values of these predictions appropriately, we regularize the posterior distribution of the labels towards a reference distribution. We show that our framework generalizes well to three different domains with 50%-68% mean absolute error reduction than the current state-of-the-art system trained for news sentence specificity. We also demonstrate the potential of our work in improving the quality and informativeness of dialogue generation systems.

Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation

“…Prediction of sentence specificity following the dictionary definition and pragmatically cast as "level of detail" was first proposed by Louis and Nenkova (2011), who related specificity to discourse relations. Sentence specificity predictors have since been developed (Louis and Nenkova, 2011;Li and Nenkova, 2015;Lugini and Litman, 2017;Ko et al, 2019). Insights from these featurerich systems and hand-code analysis (Li et al, 2016e) showed that sentence specificity encompasses multiple phenomena, including referring expressions, concreteness of concepts, gradable adjectives, subjectivity and syntactic structure.…”

“…Linguistically-informed specificity We use the system developed by Ko et al (2019), which estimates specificity as a real value. This system adopts a pragmatic notion of specificity-level of details in text-that is originally derived using sentence pairs connected via the INSTANTIATION discourse relation (Louis and Nenkova, 2011).…”

“…We use this particular system since other specificity predictors are trained on news with binary specificity labels (Li and Nenkova, 2015). Ko et al (2019) is an unsupervised domain adaptation system that predicts continuous specificity values, and was evaluated to be close to human judgments across several domains. We retrain their system using the gold responses in our data as unlabeled sentences in the unsupervised domain adaptation component.…”

“…At test time, we set the specificity level to its maximum value to force specific responses, which we found to be most beneficial. We integrate linguistic (Ko et al, 2019), information-theoretic, and frequency-based specificity metrics to better understand their roles in guiding response generation.…”

Adaptive Ensembling: Unsupervised Domain Adaptation for Political Document Analysis

“…This has gained popularity in NLP (Zhang et al, 2017;Fu et al, 2017;Chen et al, 2018), however the difficulties of adversarial training are well-established (Salimans et al, 2016;Arjovsky and Bottou, 2017). Consistency regularization methods (e.g., self-ensembling) outperform adversarial methods on visual semi-supervised and domain adaptation tasks (Athiwaratkun et al, 2019), but have rarely been applied to textual data (Ko et al, 2019). Finally, Huang and Paul (2018) establish the feasibility of using domain adaptation to label documents from discrete time periods.…”