Fair and Argumentative Language Modeling for Computational Argumentation

Holtermann, Carolin; Lauscher, Anne; Ponzetto, Simone Paolo

doi:10.18653/v1/2022.acl-long.541

Cited by 3 publications

(3 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We can clearly observe a catastrophic drop in smoothed entropy for beam and greedy search whereas smoothed entropy of well-tuned samplingbased decoding algorithms stays mostly within the stable entropy zone. These stochastic decoding algorithms are also known to produce better completions (Holtermann et al, 2022). We postulate that these two things might be related.…”

Section: Stable Entropy Hypothesismentioning

confidence: 91%

The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generation

Arora¹,

O’Donnell²,

Precup³

et al. 2023

Preprint

View full text Add to dashboard Cite

State-of-the-art language generation models can degenerate when applied to open-ended generation problems such as text completion, story generation, or dialog modeling. This degeneration usually shows up in the form of incoherence, lack of vocabulary diversity, and self-repetition or copying from the context. In this paper, we postulate that "human-like" generations usually lie in a narrow and nearly flat entropy band, and violation of these entropy bounds correlates with degenerate behavior. Our experiments show that this stable narrow entropy zone exists across models, tasks, and domains and confirm the hypothesis that violations of this zone correlate with degeneration. We then use this insight to propose an entropy-aware decoding algorithm that respects these entropy bounds resulting in less degenerate, more contextual, and "human-like" language generation in open-ended text generation settings. 1

show abstract

Section: Stable Entropy Hypothesismentioning

confidence: 91%

The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generation

Arora¹,

O’Donnell²,

Precup³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We experiment with all models described in §3.2.1 in a fine-tuning setup. Following established work on argument scoring (e.g., Gretz et al, 2020;Holtermann et al, 2022), we concatenate the description d with the action a using a separator token (e.g.,…”

Section: Canonical Rebuttal Scoringmentioning

confidence: 99%

Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals

Purkayastha,

Lauscher,

Gurevych

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

In many domains of argumentation, people's arguments are driven by so-called attitude roots, i.e., underlying beliefs and world views, and their corresponding attitude themes. Given the strength of these latent drivers of arguments, recent work in psychology suggests that instead of directly countering surface-level reasoning (e.g., falsifying given premises), one should follow an argumentation style inspired by the Jiu-Jitsu "soft" combat system (Hornsey and Fielding, 2017): first, identify an arguer's attitude roots and themes, and then choose a prototypical rebuttal that is aligned with those drivers instead of invalidating those. In this work, we are the first to explore Jiu-Jitsu argumentation for peer review by proposing the novel task of attitude and theme-guided rebuttal generation.To this end, we enrich an existing dataset for discourse structure in peer reviews with attitude roots, attitude themes, and canonical rebuttals. To facilitate this process, we recast established annotation concepts from the domain of peer reviews (e.g., aspects a review sentence is relating to) and train domain-specific models. We then propose strong rebuttal generation strategies, which we benchmark on our novel dataset for the task of end-to-end attitude and themeguided rebuttal generation and two subtasks. 1

show abstract

“…Intermediate language modeling on texts from the same or similar distribution as the downstream data has been shown to lead to improvements on various NLP tasks (e.g., Gururangan et al, 2020). During this process, the goal is to inject additional information into the PLM and thus specialize the model for a particular domain (e.g., Aharoni and Goldberg, 2020;Hung et al, 2022a;Bombieri et al, 2023) or language (e.g., Glavaš et al, 2020) or to encode other types of knowledge such as common sense knowledge (e.g., Lauscher et al, 2020a), argumentation knowledge (e.g., Holtermann et al, 2022), or geographic knowledge (e.g., Hofmann et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers

Hung,

Lauscher,

Hovy

et al. 2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

Demographic factors (e.g., gender or age) shape our language. Previous work showed that incorporating demographic factors can consistently improve performance for various NLP tasks with traditional NLP models. In this work, we investigate whether these previous findings still hold with state-of-the-art pretrained Transformer-based language models (PLMs). We use three common specialization methods proven effective for incorporating external knowledge into pretrained Transformers (e.g., domain-specific or geographic knowledge). We adapt the language representations for the demographic dimensions of gender and age, using continuous language modeling and dynamic multi-task learning for adaptation, where we couple language modeling objectives with the prediction of demographic classes. Our results, when employing a multilingual PLM, show substantial gains in task performance across four languages (English, German, French, and Danish), which is consistent with the results of previous work. However, controlling for confounding factors -primarily domain and language proficiency of Transformerbased PLMs -shows that downstream performance gains from our demographic adaptation do not actually stem from demographic knowledge. Our results indicate that demographic specialization of PLMs, while holding promise for positive societal impact, still represents an unsolved problem for (modern) NLP.

show abstract

Fair and Argumentative Language Modeling for Computational Argumentation

Cited by 3 publications

References 36 publications

The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generation

The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generation

Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals

Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers

Contact Info

Product

Resources

About