A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges

Abstract

Legal judgment prediction (LJP) applies Natural Language Processing (NLP) techniques to predict judgment results based on fact descriptions automatically. The present work addresses the growing interest in the application of NLP techniques to the task of LJP. Despite the current performance gap between machines and humans, promising results have been achieved in a variety of benchmark datasets, owing to recent advances in NLP research and the availability of large-scale public datasets. To provide a comprehensive survey of existing LJP tasks, datasets, models, and evaluations, this study presents the following contributions: (1) an analysis of 43 LJP datasets constructed in 9 different languages, together with a classification method of LJP based on three different attributes; (2) a summary of 16 evaluation metrics categorized into 4 different types to evaluate the performance of LJP models for different outputs;(3) a review of 8 legal-domain pretrained models in 4 languages, highlighting four major research directions for LJP; (4) state-of-the-art results for 11 representative datasets from different court cases and an in-depth discussion of the open challenges in this area. This study aims to provide a comprehensive review for NLP researchers and legal professionals to understand the advances in LJP over the past years, and to facilitate further joint efforts towards improving the performance of LJP models.

Predicting the Amount of Compensation for Harm Awarded by Courts Using Machine-Learning Algorithms

“…When it comes to more recent literature, US Supreme Court decisions have been analysed (Katz et al, 2017), along with the European Court of Human Rights case law (Aletras et al, 2016;Medvedeva et al, 2020;Valvoda, Cotterell & Teufel, 2023) and judgements pronounced in France (Sulea et al, 2017), Germany (Waltl et al, 2017), the Philippines (Virtucio et al, 2018), UK (Strickson & La Iglesia, 2020), Turkey (Mumcuoğlu et al, 2021). Also noteworthy are the latest extensive literature reviews on the subject by Cui, Shen & Wen (2023) and by Medvedeva, Wieling & Vols (2023).…”

Large language models for automated Q&A involving legal documents: a survey on algorithms, frameworks and applications

“…LJP is the application of NLP technology to automatically predict judgment results based on factual descriptions. Cui et al (2023a) conducted a comprehensive survey on the existing LJP tasks, data sets, models and evaluation standards. First, LJP data sets constructed in different languages were analyzed and summarized into LJP classification methods based on three different attributes.…”

LPN: Label-Enhanced Prototypical Network for Legal Judgment Prediction

“…(3) Units that commit the crimes mentioned in (1) and (2) shall be fined, where the directly responsible managers and other directly responsible personnel shall be punished following the provisions in (1).…”

“…Legal judgment prediction (LJP) refers to predicting the judgment result based on the factual description of cases and established statutes [1]. Depending on the results, it is usually divided into three subtasks, i.e., relevant article prediction, charge prediction, and penalty term prediction.…”