LILA: A Unified Benchmark for Mathematical Reasoning

Mishra, Swaroop; Finlayson, Matthew; Lü, Ping; Tang, Leonard; Welleck, Sean; Baral, Chitta; Rajpurohit, Tanmay; Tafjord, Oyvind; Sabharwal, Ashish; Clark, Peter E.; Kalyan, Ashwin

doi:10.18653/v1/2022.emnlp-main.392

Cited by 12 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…NeuLR (Xu et al, 2023c) assesses deductive reasoning, inductive reasoning, and abductive reasoning, emphasizing LLMs' capabilities in these distinct reasoning directions. TabMWP (Lu et al, 2023b), LILA (Mishra et al, 2022a), and miniF2F v1 (Zheng et al, 2022) all scrutinize LLMs' reasoning prowess in mathematics. The TabMWP dataset requires LLMs to engage in table-based Q&A and mathematical reasoning based on provided text and table data.…”

Section: Reasoningmentioning

confidence: 99%

See 1 more Smart Citation

Datasets for Large Language Models: A Comprehensive Survey

Liu,

Cao,

Liu

et al. 2024

Preprint

View full text Add to dashboard Cite

This paper embarks on an exploration into the Large Language Model (LLM) datasets, which play a crucial role in the remarkable advancements of LLMs. The datasets serve as the foundational infrastructure analogous to a root system that sustains and nurtures the development of LLMs. Consequently, examination of these datasets emerges as a critical topic in research. In order to address the current lack of a comprehensive overview and thorough analysis of LLM datasets, and to gain insights into their current status and future trends, this survey consolidates and categorizes the fundamental aspects of LLM datasets from five perspectives: (1) Pre-training Corpora; (2) Instruction Fine-tuning Datasets; (3) Preference Datasets; (4) Evaluation Datasets; (5) Traditional Natural Language Processing (NLP) Datasets. The survey sheds light on the prevailing challenges and points out potential avenues for future investigation. Additionally, a comprehensive review of the existing available dataset resources is also provided, including statistics from 444 datasets, covering 8 language categories and spanning 32 domains. Information from 20 dimensions is incorporated into the dataset statistics. The total data size surveyed surpasses 774.5 TB for pre-training corpora and 700M instances for other datasets. We aim to present the entire landscape of LLM text datasets, serving as a comprehensive reference for researchers in this field and contributing to future studies. Related resources are available at: \href{https://github.com/lmmlzn/Awesome-LLMs-Datasets}{https://github.com/lmmlzn/Awesome-LLMs-Datasets}.

show abstract

Section: Reasoningmentioning

confidence: 99%

“…Models are tasked with selecting the optimal option from two alternatives within predefined scenarios. • LILA (Mishra et al, 2022a). The LILA dataset evaluates LLMs' mathematical reasoning skills through 23 tasks across four dimensions.…”

Section: D5 Reasoningmentioning

confidence: 99%

Datasets for Large Language Models: A Comprehensive Survey

Liu,

Cao,

Liu

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Geometry3K (Lu et al, 2021) is a geometry problem-solving dataset that provides formal representations, but the dataset size is small and the problems do not require complex reasoning. AQuA (Ling et al, 2017), NumGLUE (Mishra et al, 2022b) and Lila (Mishra et al, 2022a) are large-scale datasets of various math problems. They have been used as benchmarks in solving math word problems and mathematical reasoning tasks, but we find that these datasets require only a few reasoning steps.…”

Section: Related Workmentioning

confidence: 99%

Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset

Wu,

Hui,

Chen

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Mathematical understanding and reasoning are crucial tasks for assessing the capabilities of artificial intelligence (AI). However, existing benchmarks either require just a few steps of reasoning, or only contain a small amount of data in one specific topic, making it hard to analyse AI's behaviour with reference to different problems within a specific topic in detail. In this work, we propose CONIC10K, a challenging math problem dataset on conic sections in Chinese senior high school education. Our dataset contains various problems with different reasoning depths, while only the knowledge from conic sections is required. Since the dataset only involves a narrow range of knowledge, it is easy to separately analyse the knowledge a model possesses and the reasoning ability it has. For each problem, we provide a high-quality formal representation, the reasoning steps, and the final solution. Experiments show that existing large language models, including GPT-4, exhibit weak performance on complex reasoning. We hope that our findings could inspire more advanced techniques for precise natural language understanding and reasoning. Our dataset and codes are available at https://github.com/whyNLP/Conic10K.

show abstract

“…For example, chain of thought (Wei et al, 2022) and scratchpad (Nye et al, 2021) induce generation of explanations associated with a reasoning question. Similarly other methods induces specific reasoning structures such as question summarization (Kuznia et al, 2022), question decomposition (Patel et al, 2022), program generation (Mishra et al, 2022a;Chen et al, 2022;Gao et al, 2023b), etc. However, in a real world user traffic, queries can be diverse covering various reasoning structures.…”

Section: Reasoning and Planningmentioning

confidence: 99%

Glutamate dehydrogenase 1 mediated glutaminolysis sustains HCC cells proliferation and survival under glucose deprivation

Zhou¹,

Yu²,

Cheng³

et al. 2020

Preprint

View full text Add to dashboard Cite

Background: It is generally believed that tumor cells could sustain its proliferation and survival under different nutrient status according to a so-called metabolic flexibility. How the metabolic flexibility of glutamine metabolism of HCC cells behaves under different glucose conditions has not yet been fully elucidated. In this study, we investigated how the glutamine metabolism modulate the proliferation and survival of HCC cells in response to different glucose conditions and explored the underlying molecular mechanism.Methods: Two cell lines SK-Hep-1 and PLC/PRF/5 were used to evaluate the glutamine addiction of HCC cells. Then, the cells were cultivated in high glucose medium (25mM glucose) and low glucose medium (1.0 mM glucose), respectively, to investigate whether glutaminolysis changed in response to different glucose levels. And, the underlying mechanism of glutamate dehydrogenase 1 (GDH1) sustaining HCC cells survival under glucose deprivation was explored. Additionally, the underlying correlation of GDH1 and glutamate–oxaloacetate transaminase 1 (GOT1) in glucose -poor HCC tissue was investigated.Results: HCC cells were addicted to glutamine. The glutaminolysis of HCC cells was different in response to different glucose conditions. That is, glutamate transaminases GOT1 involved glutamine metabolism played a dominant role in regulating cell growth when glucose was sufficient, while deaminase GDH1 mediated glutaminolysis became dominant when glucose was limited. Mechanically, low-glucose treated HCC cells could induce an elevated expression of GDH1 to supplement the TCA cycles in respond to glucose deprivation. Additionally, we further uncovered an underlying negative association between GDH1 and GOT1 in HCC tissues with decreased glucose levelsConclusions: GDH1 mediated pathway played a leading role in maintaining cell proliferation and survival under low glucose condition. By contrast, GOT1 mediated pathway was activated under high glucose condition. Mechanically, highly expressed GDH1 could drive the TCA cycle in response to glucose deprivation. Besides, there was a potential negative correlation between GDH1 and GOT1 in glucose-poor HCC tissues.

show abstract

LILA: A Unified Benchmark for Mathematical Reasoning

Cited by 12 publications

References 0 publications

Datasets for Large Language Models: A Comprehensive Survey

Datasets for Large Language Models: A Comprehensive Survey

Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset

Glutamate dehydrogenase 1 mediated glutaminolysis sustains HCC cells proliferation and survival under glucose deprivation

Contact Info

Product

Resources

About