Standing on the Shoulders of Giant Frozen Language Models

Abstract

Huge pretrained language models (LMs) have demonstrated surprisingly good zero-shot capabilities on a wide variety of tasks. This gives rise to the appealing vision of a single, versatile model with a wide range of functionalities across disparate applications. However, current leading techniques for leveraging a "frozen" LM-i.e., leaving its weights untouched-still often underperform fine-tuning approaches which modify these weights in a task-dependent way. Those, in turn, suffer forgetfulness and compromise versatility, suggesting a tradeoff between performance and versatility. The main message of this paper is that current frozenmodel techniques such as prompt tuning are only the tip of the iceberg, and more powerful methods for leveraging frozen LMs can do just as well as fine tuning in challenging domains without sacrificing the underlying model's versatility. To demonstrate this, we introduce three novel methods for leveraging frozen models: input-dependent prompt tuning, frozen readers, and recursive LMs, each of which vastly improves on current frozen-model approaches. Indeed, some of our methods even outperform fine-tuning approaches in domains currently dominated by the latter. The computational cost of each method is higher than that of existing frozen model methods, but still negligible relative to a single pass through a huge frozen LM. Each of these methods constitutes a meaningful contribution in its own right, but by presenting these contributions together we aim to convince the reader of a broader message that goes beyond the details of any given method: that frozen models have untapped potential and that fine-tuning is often unnecessary.

Geography of Technology Transfer in China

“…Recent studies in Transformerbased language models such as ELMo [15] and BERT [4] have shown their capabilities in scaling up model sizes with pre-training methodology such as Masked Language Modeling [4]. Shortly after, several Large Language Models (LLMs), e.g., the GPT family [1,13], PaLM [2], Jurassic-X [9], Megatron-Turing [16] , LaMDA [17],LLaMA [18], have been emerging with huge amount of parameters of up to 100B-5000B parameters. They have shown great advantages in language modeling tasks, such as arithmetic reasoning, commonsense reasoning, symbolic reasoning and natural language inference.…”

AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning

“…Workflows are often more expensive than simpler approaches (e.g., Chain of Thought prompting) [35,38,45,53,62,90,104,105,110,134,150,157]. However, the cost is often less than hiring professionals [6,30,43,89,151], pretraining models [31,57,69,86,150], or completing the task yourself [135,156].…”