Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling

Abstract

Given the claims of improved text generation quality across various pre-trained neural models, we consider the coherence evaluation of machine generated text to be one of the principal applications of coherence models that needs to be investigated. Prior work in neural coherence modeling has primarily focused on devising new architectures for solving the permuted document task. We instead use a basic model architecture and show significant improvements over state of the art within the same training regime. We then design a harder self-supervision objective by increasing the ratio of negative samples within a contrastive learning setup, and enhance the model further through automatic hard negative mining coupled with a large global negative queue encoded by a momentum encoder. We show empirically that increasing the density of negative samples improves the basic model, and using a global negative queue further improves and stabilizes the model while training with hard negative samples. We evaluate the coherence model on task-independent test sets that resemble real-world applications and show significant improvements in coherence evaluations of downstream tasks. 1

The Adoption and Efficacy of Large Language Models: Evidence From Consumer Complaints in the Financial Industry

“…Our analyses reveal that Likely-AI complaints were significantly different from Likely-Human complaints across various linguistic features. The first column of Table 2 shows that Likely-AI complaints were more coherent (i.e., show higher coherence scores) than Likely-Human complaints based on the recently developed coherence metric (Jwalapuram, Joty, and Lin, 2022). Moreover, as shown in the second column of Table 2, Likely-AI complaints were easier to read than Likely-Human complaints, as lower readability scores (Kincaid, Fishburne, Rogers, and Chissom, 1975) indicate texts that are easier to read.…”

“…We compared these two groups in terms of linguistic features proposed by prior computational linguistics research (Kincaid, Fishburne, Rogers, and Chissom, 1975;Jwalapuram, Joty, and Lin, 2022;Danescu-Niculescu-Mizil, Sudhof, Jurafsky, Leskovec, and Potts, 2013;Huang, Wang, and Yang, 2023). Each column in Table 2 presents results from a regression analysis in which the dependent variable was each computational linguistic metric estimated for the complaints and the independent variable was the binary variable of whether the complaint was Likely-AI (1) or Likely-Human (0) as in Table 1.…”

Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators

“…On the other hand, the paragraph-level coherence score is determined by the normalized score of a discourse coherence model (Jwalapuram et al, 2021), denoted as S coh_para (k):…”

Aspect-Category Enhanced Learning with a Neural Coherence Model for Implicit Sentiment Analysis

“…text summarization, text classification, and text quality assessment. It is also utilized to model the coherence representation (Jwalapuram et al, 2022), because it takes advantage of both the autoregressive model and the BERT (Devlin et al, 2019). Formally, for a sequence x of the length I, there are I!…”

“…Document-Level Coherence with Contrastive Learning. Following Jwalapuram et al (2022), to learn robust coherence representations, we adopt the sentence ordering task by using contrastive learning. It enforces that the coherence score of the positive sample (original document) should be higher than that of the negative sample (disorder document).…”