Language Model Crossover: Variation through Few-Shot Prompting

Abstract

This paper pursues the insight that language models naturally enable an intelligent variation operator similar in spirit to evolutionary crossover. In particular, language models of sufficient scale demonstrate in-context learning, i.e. they can learn from associations between a small number of input patterns to generate outputs incorporating such associations (also called few-shot prompting). This ability can be leveraged to form a simple but powerful variation operator, i.e. to prompt a language model with a few text-based genotypes (such as code, plain-text sentences, or equations), and to parse its corresponding output as those genotypes' offspring. The promise of such language model crossover (which is simple to implement and can leverage many different open-source language models) is that it enables a simple mechanism to evolve semantically-rich text representations (with few domain-specific tweaks), and naturally benefits from current progress in language models. Experiments in this paper highlight the versatility of language-model crossover, through evolving binary bit-strings, sentences, equations, text-toimage prompts, and Python code. The conclusion is that language model crossover is a promising method for evolving genomes representable as text. CCS CONCEPTS• Computing methodologies → Neural networks.

Tailoring Education with GenAI: A New Horizon in Lesson Planning

“…In this approach, the GenAI tools takes this text as input and generates the answer or ranks different options [20] in contrast to the zero-shot prompting, which involves text generation without training on the specific task [21]. This method can be more scalable and sometimes requires relatively few input data [22]. Although this method can be considered a common way to help genAI models understand and cope with a task, even this method can be further improved.…”

Generative Calibration for In-context Learning

“…It is shown that the in-context model would imitate the prompting examples to generate similar sequences (Meyerson et al, 2023). One principled and popular explanation is that such a model is an approximation of the posterior predictive distribution in Bayesian statistics (Xie et al, 2021;:…”