Solving Quantitative Reasoning Problems with Language Models

Lewkowycz, Aitor; Andreassen, Anders; Dohan, D.; Dyer, Ethan; Michalewski, Henryk; Ramasesh, Vinay; Slone, Ambrose; Anil, Cem; Schlag, Imanol; Gutman-Solo, Theo; Wu, Yuhuai; Neyshabur, Behnam; Gur-Ari, Guy; Misra, Vedant

doi:10.48550/arxiv.2206.14858

Cited by 48 publications

(64 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Fig. 6, we take datasets sampled from our generative data model, (19) and (23), and map them through our random feature model, (28), for a fixed set of features weights, u, in order to verify that their spectra has the properties discussed in §2.2. We see that the power-law portion of each spectrum is controlled by the minimum of the number of features, N , and the size of the dataset, T .…”

Section: Random Feature Modelmentioning

confidence: 99%

“…After such a study, a number of follow ups appeared showing even more general applicability and more detailed understanding [20][21][22][23] -and even improved performance scaling with data size from a power law to an exponential falloff with clever pruning [24]. 1 At the same time, autoregressive generative modeling with transformers has continued to be applied to broader AI tasks such as coding [27], quantitative reasoning [28], and even on the suite of computer vision tasks, with the advent of the Vision Transformer (ViT) [29] family of models.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Solvable Model of Neural Scaling Laws

Maloney¹,

Roberts²,

Sully³

2022

Preprint

View full text Add to dashboard Cite

Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws: specifically, their performance behaves predictably as a power law in either parameters or dataset size until bottlenecked by the other resource. To understand this better, we first identify the necessary properties allowing such scaling laws to arise and then propose a statistical model -a joint generative data model and random feature model -that captures this neural scaling phenomenology. By solving this model in the dual limit of large training set size and large number of parameters, we gain insight into (i) the statistical structure of datasets and tasks that lead to scaling laws, (ii) the way nonlinear feature maps, such as those provided by neural networks, enable scaling laws when trained on these datasets, (iii) the optimality of the equiparameterization scaling of training sets and parameters, and (iv) whether such scaling laws can break down and how they behave when they do. Key findings are the manner in which the power laws that occur in the statistics of natural datasets are extended by nonlinear random feature maps and then translated into power-law scalings of the test loss and how the finite extent of the data's spectral power law causes the model's performance to plateau.Equal contribution.

show abstract

Section: Random Feature Modelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Solvable Model of Neural Scaling Laws

Maloney¹,

Roberts²,

Sully³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Pre-trained models, such as BERT [19], can be fine-tuned to achieve high performance on downstream tasks such as sentiment analysis. Minerva [20], a recent language model, has shown promising performance on even more complex tasks such as undergraduate-level STEM problems. Pre-training on large amounts of unlabeled sequences also holds a promise for biology.…”

Section: Introductionmentioning

confidence: 99%

DNA language models are powerful predictors of genome-wide variant effects

Benegas

Batra

Song

2022

Preprint

View full text Add to dashboard Cite

Variant effect prediction has traditionally focused on training supervised models on labeled data. Motivated by recent advances in natural language processing that have demonstrated substantial gains on diverse tasks by pre-training on large unlabeled data, however, unsupervised pre-training on massive databases of protein sequences has proven to be an effective approach to extracting complex information about proteins. Such models have been shown to learn variant effects in coding regions in a zero-shot manner. In a similar vein, we here introduce GPN (Genomic Pre-trained Network) which can learn variant effects in non-coding DNA using unsupervised pre-training on genomic DNA sequence alone. Our model is also able to learn gene structure and DNA motifs without any supervision. We demonstrate the utility of GPN by showing that it outperforms supervised deep learning models such as DeepSEA trained on vast amounts of functional genomics data in Arabidopsis thaliana, a model organism for plant biology. Additionally, GPN trained on a single genome outperforms popular conservation scores such as phyloP and PhastCons, which are computed using aligned genomes from multiple species and can be used to predict the pathogenicity of variants that perturb highly conserved positions. We provide code (https://github.com/songlab-cal/gpn) to train GPN for any given species using its DNA sequence alone and learn corresponding zero-shot variant effects genome-wide.

show abstract

“…Machine learning tools designed for text prediction [56,57] can be fine-tuned to "auto-complete" mathematical proofs, given a formal problem statement [34,58], even to the point of generating correct solutions to International Math Olympiad problems [59]. Recently, large language models have demonstrated capabilities in solving chemistry problems [60,61], as well as answering scientific question-and-answer problems invoking quantitative reasoning [62] Formal proofs in science and engineering mathematics may in the future provide useful, high-quality data for artificial intelligences aiming to learn, reason, and discover in science [63,64,65].…”

Section: Discussionmentioning

confidence: 99%

Formalizing Chemical Theory using the Lean Theorem Prover

Bobbin¹,

Sharlin²,

Feyzishendi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Chemical theory can be made more rigorous using the Lean theorem prover, an interactive theorem prover for complex mathematics. We formalize the Langmuir [1] and BET [2] theories of adsorption, making each scientific premise clear and every step of the derivations explicit. Lean's math library, mathlib, provides formally-verified theorems for infinite geometries series, which are central to BET theory. While writing these proofs, Lean prompts us to include mathematical constraints that were not originally reported. We also illustrate how Lean flexibly enables the reuse of proofs that build on more complex theories through the use of functions, definitions, and structures. Finally, we construct scientific frameworks for interoperable proofs, by creating structures for classical thermodynamics and kinematics, using them to formalize gas law relationships like Boyle's Law and equations of motion underlying Newtonian mechanics, respectively. This approach can be extended to other fields, enabling the formalization of rich and complex theories in science and engineering.

show abstract

Solving Quantitative Reasoning Problems with Language Models

Cited by 48 publications

References 16 publications

A Solvable Model of Neural Scaling Laws

A Solvable Model of Neural Scaling Laws

DNA language models are powerful predictors of genome-wide variant effects

Formalizing Chemical Theory using the Lean Theorem Prover

Contact Info

Product

Resources

About