Human Evaluation Experiment of Legal Information Retrieval Methods

Novotná, Tereza

doi:10.3233/faia210328

Cited by 1 publication

(1 citation statement)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Depending on the application, the queries can be short (e.g., simple keywords), of medium length (e.g., Boolean or natural language queries) or long (e.g., whole documents). Early legal retrieval systems are generally unsupervised [86] while recent research tends to center around different flavors of machine learning techniques [87], [88]. By using word embeddings, several systems have been developed to find the relevant sentences within specific documents [89], [90].…”

Section: A An Overview Of Major Legal Nlp Tasksmentioning

confidence: 99%

On the Effectiveness of Pre-Trained Language Models for Legal Natural Language Processing: An Empirical Study

Song

Gao

et al. 2022

IEEE Access

View full text Add to dashboard Cite

We present the first comprehensive empirical evaluation of pre-trained language models (PLMs) for legal natural language processing (NLP) in order to examine their effectiveness in this domain.Our study covers eight representative and challenging legal datasets, ranging from 900 to 57K samples, across five NLP tasks: binary classification, multi-label classification, multiple choice question answering, summarization and information retrieval. We first run unsupervised, classical machine learning and/or non-PLM based deep learning methods on these datasets, and show that baseline systems' performance can be 4%∼35% lower than that of PLM-based methods. Next, we compare general-domain PLMs and those specifically pre-trained for the legal domain, and find that domain-specific PLMs demonstrate 1%∼5% higher performance than general-domain models, but only when the datasets are extremely close to the pretraining corpora. Finally, we evaluate six general-domain state-of-the-art systems, and show that they have limited generalizability to legal data, with performance gains from 0.1% to 1.2% over other PLM-based methods. Our experiments suggest that both general-domain and domain-specific PLM-based methods generally achieve better results than simpler methods on most tasks, with the exception of the retrieval task, where the best-performing baseline outperformed all PLM-based methods by at least 5%. Our findings can help legal NLP practitioners choose the appropriate methods for different tasks, and also shed light on potential future directions for legal NLP research.

show abstract

Section: A An Overview Of Major Legal Nlp Tasksmentioning

confidence: 99%