Refocusing on Relevance: Personalization in NLG

Abstract

Many NLG tasks such as summarization, dialogue response, or open domain question answering focus primarily on a source text in order to generate a target response. This standard approach falls short, however, when a user's intent or context of work is not easily recoverable based solely on that source texta scenario that we argue is more of the rule than the exception. In this work, we argue that NLG systems in general should place a much higher level of emphasis on making use of additional context, and suggest that relevance (as used in Information Retrieval) be thought of as a crucial tool for designing user-oriented text-generating tasks. We further discuss possible harms and hazards around such personalization, and argue that value-sensitive design represents a crucial path forward through these challenges.

DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4

“…It is critical for the output from LLMs to respond to a variety of user attributes, ranging from interaction history to the situation of use. In the past, researchers have investigated user modeling for various tasks, including headline generation, dialog response generation and recipe creation (Majumder et al, 2019;Flek, 2020;Wu et al, 2021;Dudy et al, 2021;Cai et al, 2023). In this paper, our focus has been on exploring human preference modeling for LLM development.…”

Book Review: Reading Computer-Generated Texts

“…Natural language generation (NLG) is an interesting application of artificial intelligence which focuses exclusively on the automatic production of understandable written or spoken language. To date, numerous studies have been conducted to explore the design and implementation of NLG systems (Singh et al, 2016), the system evaluation (Van der Lee et al, 2021), and the personalization of NLG systems (Dudy et al, 2021). Reading Computer-generated Text takes a step further by examining NLG from a humanities perspective.…”