Evolution through Large Models

Lehman, Joel; Gordon, Jonathan; Jain, Shawn; Kamal, Ndousse,; Yeh, Cathy; Stanley, Kenneth O.

doi:10.48550/arxiv.2206.08896

Cited by 8 publications

(16 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This work builds on previous results that highlight the potential of LMs to generate [69] or augment [26] data, although here the focus is on enabling continual evolution. It also is closely related to work demonstrating that LMs can be trained to embody intelligent mutation operators for code [46], i.e. by training a model on changes to code files gathered from GitHub; Lehman et al [46] also proposed a way to generate domain-specific mutations from hand-designed prompts for LMs.…”

Section: Intelligent Variation Operatorsmentioning

confidence: 99%

“…It also is closely related to work demonstrating that LMs can be trained to embody intelligent mutation operators for code [46], i.e. by training a model on changes to code files gathered from GitHub; Lehman et al [46] also proposed a way to generate domain-specific mutations from hand-designed prompts for LMs. Such operators may require fine-tuning a model on mutation-like training data gathered from GitHub or hand-specifying example mutations, and have been applied only to the domain of code.…”

Section: Intelligent Variation Operatorsmentioning

confidence: 99%

“…To address this limitation, strategies for generating intelligent variation have been a focus of much research in EC. For example, evolving within a latent space of an ML model [24,71], or through training models to mimic mutations [37,46]. One particularly popular such strategy is to build probabilistic models of high-performing individuals or to model elements of the search path taken across recent generations.…”

Section: Intelligent Variation Operatorsmentioning

confidence: 99%

“…from text to image). Interestingly, LMX may also synergize with recent LM-based mutation techniques [46], and is amenable to similar possibilities such as fine-tuning an LM as a way of accelerating search, although we leave these possibilities for future work.…”

Section: Abstract 1 Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Language Model Crossover: Variation through Few-Shot Prompting

Meyerson¹,

Nelson²,

Bradley³

et al. 2023

Preprint

View full text Add to dashboard Cite

This paper pursues the insight that language models naturally enable an intelligent variation operator similar in spirit to evolutionary crossover. In particular, language models of sufficient scale demonstrate in-context learning, i.e. they can learn from associations between a small number of input patterns to generate outputs incorporating such associations (also called few-shot prompting). This ability can be leveraged to form a simple but powerful variation operator, i.e. to prompt a language model with a few text-based genotypes (such as code, plain-text sentences, or equations), and to parse its corresponding output as those genotypes' offspring. The promise of such language model crossover (which is simple to implement and can leverage many different open-source language models) is that it enables a simple mechanism to evolve semantically-rich text representations (with few domain-specific tweaks), and naturally benefits from current progress in language models. Experiments in this paper highlight the versatility of language-model crossover, through evolving binary bit-strings, sentences, equations, text-toimage prompts, and Python code. The conclusion is that language model crossover is a promising method for evolving genomes representable as text. CCS CONCEPTS• Computing methodologies → Neural networks.

show abstract

Section: Intelligent Variation Operatorsmentioning

confidence: 99%

Section: Intelligent Variation Operatorsmentioning

confidence: 99%

Section: Intelligent Variation Operatorsmentioning

confidence: 99%

Section: Abstract 1 Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Language Model Crossover: Variation through Few-Shot Prompting

Meyerson¹,

Nelson²,

Bradley³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…LLM can be viewed as a newer metalearning case. 249 Jane X Wang of DeepMind team published a review on metalearning in natural learning and AI. 250…”

Section: Anns and Natural Rule Typesmentioning

confidence: 99%

Evolving Drug Design Methodology: from QSAR to AIDD

2022

Preprint

View full text Add to dashboard Cite

When medicinal chemistry was born a hundred years ago, a drug design methodology was expected to be based on the knowledge of the relations among chemistry, biology and medicine. Originally, chemists believed that a drug molecule consists of a scaffold with several substituents. While the substituents were replaced by alternate functional groups (aka substructures), the activity value of the molecule would be changed accordingly. This is termed as structure–activity relationship (SAR), which can be used to guide chemists to chemically modify the molecule to improve its druggability. Along with the progress of computing technology, SAR evolved into QSAR (Quantitative SAR). QSAR method prevailed in the era when determinism dominated the scientific community. Therefore, the paradigm of QSAR studies was based on the thought of “discovering the analytical rules (analytical formula of functional) between independent variables and functions hidden in experimental data by curve fitting or regression”. Earlier QSAR was based upon so called the similarity and additivity postulates. With the advent of the era of high-throughput experiments and big data, the two postulates are facing serious challenges. Coupled with the puzzling problems (such as substructure partitioning, “activity cliff”, unbalanced data sampling, and the paradox of prediction accuracy and generalization), conventional QSAR was declining from mature. In the beginning of this century, artificial intelligence (AI), specifically deep learning (DL), significantly succussed in image pattern recognition and natural language processing (NPL). AI was soon adopted for QSAR studies as a disruptive approach. It is now believed that drug design can be data-driven instead of rule-based (curve-fitting or regression). QSAR can also be directly revealed by AI without knowing the mechanisms of actions. Thus, the two postulates of conventional QSAR are no longer required, and the associated puzzling problems or paradoxes could be resolved. By examining the historical pathway of QSAR evolving into AI assisted drug design (AIDD), this review summarizes the process how the drug design paradigm is transformed from determinism to causalitism + probabilitism. The principles and challenges of drug design methodology are explored, the pros and cons for QSAR and AIDD are discussed with perspectives. It is worth noting that although AIDD is powerful, it is not omnipotent and should be treated rationally. The essence of machine learning is to reveal the major trends of a data set; while the minor trends (aka outliers, which are often ignored or discarded) cannot be captured by AI algorithms. However, the outliers are likely to be the entrances to disruptive discoveries. Therefore, philosophically, it is unrealistic to develop innovative drugs only relying on AIDD. AIDD’s achievements rely on inheriting the legacy of QSAR's theories, methods, technologies, and data.

show abstract

Advances of machine learning in materials science: Ideas and techniques

Sin¹,

Ng²,

Wang³

et al. 2023

Front. Phys.

View full text Add to dashboard Cite

In this big data era, the use of large dataset in conjunction with machine learning (ML) has been increasingly popular in both industry and academia. In recent times, the field of materials science is also undergoing a big data revolution, with large database and repositories appearing everywhere. Traditionally, materials science is a trial-and-error field, in both the computational and experimental departments. With the advent of machine learning-based techniques, there has been a paradigm shift: materials can now be screened quickly using ML models and even generated based on materials with similar properties; ML has also quietly infiltrated many sub-disciplinary under materials science. However, ML remains relatively new to the field and is expanding its wing quickly. There are a plethora of readily-available big data architectures and abundance of ML models and software; The call to integrate all these elements in a comprehensive research procedure is becoming an important direction of material science research. In this review, we attempt to provide an introduction and reference of ML to materials scientists, covering as much as possible the commonly used methods and applications, and discussing the future possibilities.

show abstract

Evolution through Large Models

Cited by 8 publications

References 34 publications

Language Model Crossover: Variation through Few-Shot Prompting

Language Model Crossover: Variation through Few-Shot Prompting

Evolving Drug Design Methodology: from QSAR to AIDD

Advances of machine learning in materials science: Ideas and techniques

Contact Info

Product

Resources

About