A dataset for evaluating legal question answering on private international law

Sovrano, Francesco; Palmirani, Monica; Distefano, Biagio; Sapienza, Salvatore; Vitali, Fabio

doi:10.1145/3462757.3466094

Cited by 5 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several datasets Sovrano et al (2021) and performance metrics have been proposed to evaluate the results of a given QA system. According to Rodrigo et al (Rodrigo & Peñas, 2017) the method to obtain the final score depends on the evaluation measure selected.…”

Section: Discussionmentioning

confidence: 99%

A Survey on Legal Question Answering Systems

Martinez‐Gil¹

2021

Preprint

View full text Add to dashboard Cite

Many legal professionals think that the explosion of information about local, regional, national, and international legislation makes their practice more costly, time-consuming, and even error-prone. The two main reasons for this are that most legislation is usually unstructured, and the tremendous amount and pace with which laws are released causes information overload in their daily tasks. In the case of the legal domain, the research community agrees that a system allowing to generate automatic responses to legal questions could substantially impact many practical implications in daily activities.The degree of usefulness is such that even a semi-automatic solution could significantly help to reduce the workload to be faced. This is mainly because a Question Answering system could be able to automatically process a massive amount of legal resources to answer a question or doubt in seconds, which means that it could save resources in the form of effort, money, and time to many professionals in the legal sector. In this work, we quantitatively and qualitatively survey the solutions that currently exist to meet this challenge.

show abstract

Section: Discussionmentioning

confidence: 99%

A Survey on Legal Question Answering Systems

Martinez‐Gil¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Sovrano et al (2021) released an English legal corpus and a legal knowledge exploration benchmark to facilitate the evaluation of automated Q&A methods in international private law settings. This benchmark enables detailed analyses of the performance of automated Q&A models developed for legal purposes.…”

Section: Fine-tuning Large Language Models For the Legal Domainmentioning

confidence: 99%

Large language models for automated Q&A involving legal documents: a survey on algorithms, frameworks and applications

Yang,

Wang,

Wang

et al. 2024

IJWIS

View full text Add to dashboard Cite

Purpose This study aims to adopt a systematic review approach to examine the existing literature on law and LLMs.It involves analyzing and synthesizing relevant research papers, reports and scholarly articles that discuss the use of LLMs in the legal domain. The review encompasses various aspects, including an analysis of LLMs, legal natural language processing (NLP), model tuning techniques, data processing strategies and frameworks for addressing the challenges associated with legal question-and-answer (Q&A) systems. Additionally, the study explores potential applications and services that can benefit from the integration of LLMs in the field of intelligent justice. Design/methodology/approach This paper surveys the state-of-the-art research on law LLMs and their application in the field of intelligent justice. The study aims to identify the challenges associated with developing Q&A systems based on LLMs and explores potential directions for future research and development. The ultimate goal is to contribute to the advancement of intelligent justice by effectively leveraging LLMs. Findings To effectively apply a law LLM, systematic research on LLM, legal NLP and model adjustment technology is required. Originality/value This study contributes to the field of intelligent justice by providing a comprehensive review of the current state of research on law LLMs.

show abstract

DiscoLQA: zero-shot discourse-based legal question answering on European Legislation

Sovrano,

Palmirani,

Sapienza

et al. 2024

Artif Intell Law

View full text Add to dashboard Cite

The structures of discourse used by legal and ordinary languages share differences that foster technical issues when applying or fine-tuning general-purpose language models for open-domain question answering on legal resources. For example, longer sentences may be preferred in European laws (i.e., Brussels I bis Regulation EU 1215/2012) to reduce potential ambiguities and improve comprehensibility, distracting a language model trained on ordinary English. In this article, we investigate some mechanisms to isolate and capture the discursive patterns of legalese in order to perform zero-shot question answering, i.e., without training on legal documents. Specifically, we use pre-trained open-domain answer retrieval systems and study what happens when changing the type of information to consider for retrieval. Indeed, by selecting only the important parts of discourse (e.g., elementary units of discourse, EDU for short, or abstract representations of meaning, AMR for short), we should be able to help the answer retriever identify the elements of interest. Hence, with this paper, we publish Q4EU, a new evaluation dataset that includes more than 70 questions and 200 answers on 6 different European norms, and study what happens to a baseline system when only EDUs or AMRs are used during information retrieval. Our results show that the versions using EDUs are overall the best, leading to state-of-the-art F1, precision, NDCG and MRR scores.

show abstract

A dataset for evaluating legal question answering on private international law

Cited by 5 publications

References 7 publications

A Survey on Legal Question Answering Systems

A Survey on Legal Question Answering Systems

Large language models for automated Q&A involving legal documents: a survey on algorithms, frameworks and applications

DiscoLQA: zero-shot discourse-based legal question answering on European Legislation

Contact Info

Product

Resources

About