LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

Chalkidis, Ilias; Jana, Abhik; Hartung, Dirk; Bommarito, Michael James; Androutsopoulos, Ion; Katz, Daniel; Αλέτρας, Νικόλαος

doi:10.48550/arxiv.2110.00976

Cited by 3 publications

(3 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The GLUE benchmark, encompassing a variety of natural language understanding tasks, served as a comprehensive evaluation framework, driving significant advancements in model development [17], [31]. SuperGLUE extended the original GLUE tasks with more challenging benchmarks, pushing the limits of what models could achieve in terms of understanding and reasoning [32], [33]. The HaluEval benchmark specifically targeted the evaluation of hallucinations in LLMs, providing a large-scale framework for assessing the accuracy and reliability of model outputs [7].…”

Section: B Benchmarking and Evaluation Metricsmentioning

confidence: 99%

Benchmarking the Hallucination Tendency of Google Gemini and Moonshot Kimi

Shan,

Ming,

Hong

et al. 2024

Preprint

View full text Add to dashboard Cite

To evaluate the hallucination tendencies of state-of-the-art language models is crucial for improving their reliability and applicability across various domains. This article presents a comprehensive evaluation of Google Gemini and Kimi using the HaluEval benchmark, focusing on key performance metrics such as accuracy, relevance, coherence, and hallucination rate. Google Gemini demonstrated superior performance, particularly in maintaining low hallucination rates and high contextual relevance, while Kimi, though robust, showed areas needing further refinement. The study highlights the importance of advanced training techniques and optimization in enhancing model efficiency and accuracy. Practical recommendations for future model development and optimization are provided, emphasizing the need for continuous improvement and rigorous evaluation to achieve reliable and efficient language models.

show abstract

Section: B Benchmarking and Evaluation Metricsmentioning

confidence: 99%

Benchmarking the Hallucination Tendency of Google Gemini and Moonshot Kimi

Shan,

Ming,

Hong

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…BBC News (Greene and Cunningham, 2006) and 20 NewsGroups (Lang, 1995) comprise a collection of public news articles on various topics. LEDGAR (Tuggener et al, 2020) includes a corpus of legal provisions in contract, which is part of the LexGLUE (Chalkidis et al, 2021) benchmark to evaluate the capabilities of legal text. arXiv is a digital archive that stores scholarly articles from a wide range of fields, such as mathematics, computer science, and physics.…”

Section: Datasetmentioning

confidence: 99%

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

Yun,

Kim,

Kim

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Transformer-based models have achieved dominant performance in numerous NLP tasks. Despite their remarkable successes, pre-trained transformers such as BERT suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including the ones unfavorable to classification performance. To overcome these challenges, we propose integrating two strategies: token pruning and token combining. Token pruning eliminates less important tokens in the attention mechanism's key and value as they pass through the layers. Additionally, we adopt fuzzy logic to handle uncertainty and alleviate potential mispruning risks arising from an imbalanced distribution of each token's importance. Token combining, on the other hand, condenses input sequences into smaller sizes in order to further compress the model. By integrating these two approaches, we not only improve the model's performance but also reduce its computational demands. Experiments with various datasets demonstrate superior performance compared to baseline models, especially with the best improvement over the existing BERT model, achieving +5%p in accuracy and +5.6%p in F1 score. Additionally, memory cost is reduced to 0.61x, and a speedup of 1.64x is achieved.

show abstract

“…Even though judiciary systems produce, consume and use massive volumes of textual information [9], they lack technological solutions to increase their efficiency. Moreover, legal documents are known to be complex and lengthy and use specialized vocabulary [1], which raises the technical challenge of developing NLP systems in that domain.…”

Section: Introductionmentioning

confidence: 99%

RISC: Generating Realistic Synthetic Bilingual Insurance Contract

Beauchemin

Khoury

2023

Proceedings of the Canadian Conference on Artificial Intelligence

View full text Add to dashboard Cite

This paper presents RISC, an open-source Python package data generator 1 . RISC generates look-alike automobile insurance contracts based on the Quebec regulatory insurance form in French and English. Insurance contracts are 90 to 100 pages long and use complex legal and insurance-specific vocabulary for a layperson. Hence, they are a much more complex class of documents than those in traditional NLP corpora. Therefore, we introduce RISCBAC, a Realistic Insurance Synthetic Bilingual Automobile Contract dataset based on the mandatory Quebec car insurance contract. The dataset comprises 10,000 French and English unannotated insurance contracts. RISCBAC enables NLP research for unsupervised automatic summarisation, question answering, text simplification, machine translation and more. Moreover, it can be further automatically annotated as a dataset for supervised tasks such as NER.

show abstract

LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

Cited by 3 publications

References 30 publications

Benchmarking the Hallucination Tendency of Google Gemini and Moonshot Kimi

Benchmarking the Hallucination Tendency of Google Gemini and Moonshot Kimi

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

RISC: Generating Realistic Synthetic Bilingual Insurance Contract

Contact Info

Product

Resources

About