On Conditional and Compositional Language Model Differentiable Prompting

Abstract

Prompts have been shown to be an effective method to adapt a frozen Pretrained Language Model (PLM) to perform well on downstream tasks. Prompts can be represented by a human-engineered word sequence or by a learned continuous embedding. In this work, we investigate conditional and compositional differentiable prompting. We propose a new model, Prompt Production System (ProPS), which learns to transform task instructions or input metadata, into continuous prompts that elicit task-specific outputs from the PLM. Our model uses a modular network structure based on our neural formulation of Production Systems, which allows the model to learn discrete rules -- neural functions that learn to specialize in transforming particular prompt input patterns, making it suitable for compositional transfer learning and few-shot learning. We present extensive empirical and theoretical analysis and show that ProPS consistently surpasses other PLM adaptation techniques, and often improves upon fully fine-tuned models, on compositional generalization tasks, controllable summarization and multilingual translation, while needing fewer trainable parameters.

Metaverse Counseling: Transforming Depression Care with AI Digital Humans in Virtual Realms

“…The evaluation of immersive virtual reality technology on learning and educational outcomes by [9] underscores the superiority of experiential learning through virtual reality compared to conventional educational methods. The Prompt Production System (PROPS), a new model proposed in [10], learns how to convert task prompts or input metadata into a sequence of prompts that elicit specific task outputs.…”

Benchmarking and Improving Text-to-SQL Generation under Ambiguity

“…Ambiguity in SQL Although ambiguity has been studied in other fields of NLP (Pilault et al, 2023;Futeral et al, 2022), it has been unexplored in the context of semantic parsing. Ambiguity in SQL arising from related column names is discussed in , but they only consider column ambiguity.…”