Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark

Niklaus, Joel; Chalkidis, Ilias; Stürmer, Matthias

doi:10.48550/arxiv.2110.00806

Cited by 4 publications

(13 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Language Input Output Download Rationale Decision SwissJudgmentPrediction [26] German Pleas r Facts r -Plea judgments r Link Italian French Sulea et al [14] Facts r Others r -TSCC [40] Thai Facts a Articles a -Link JUSTICE [27] English Pleas r Facts r -Plea judgments r -BStricks_LDC [41] Facts r Others r Link ILDC [42] Fact snippets r Link ACI [43] Facts a,r Charges a Link echr [13] Facts r…”

Section: Datasetmentioning

confidence: 99%

“…(2) Facts from court debate. LJP-MSJudge [54] predicts the verdicts of the plaintiffs' pleas 60,000 321 50 -RACP [48] 100,000 -51 MAMD [49] 164,997 -CPTP [50] 238,749 157 226 Court-View-Gen [33] 171,981 51 -Criminal-S [51] 77,046 149 Criminal-M [51] 191,960 149 Criminal-L [51] 383,697 149 QAjudge-CJO [52] 15,120 19 20 QAjudge-PKU [52] 14,000 19 20 QAjudge-CAIL [52] 13,423 [34] 66,904 Auto-Judge [53] 100,000 62 echr [13] ECtHR Human rights 584 3 ECHR-CASES [25] 11,478 66 ECtHR_crystal_ball [44] 11,532 14 SwissJudgmentPrediction [26] FSCS Generic 85,268 -Sulea et al [14] FSC 126,865…”

Section: Charges Rmentioning

confidence: 99%

“…In constructing datasets, there have been at least two lines of work, as seen in Table 2. The first one employs a machine to extract the necessary metadata, such as using Regular Expression to identify and mask judgment results ( [26,14,41,42]) or to extract the input-output data samples( [13,25,44,45,24,29,47,15,49,33,50,53]) from judgment documents of given court and applying the deep neural model to extract facts from Court judgment documents as [43] for evaluating the LJP model. The second line of work recruits legal experts to annotate rationale sentences [42,48] as gold sentences or to annotate legal domain knowledge (which are not included in the judgment documents, such as [46,51,52]).…”

Section: Datasetsmentioning

confidence: 99%

“…A total of 2,384 SCOTUS cases from Oyez based on a series of manually balance procedures are also evaluated to predict the judicial judgment. In addition, SwissJudgmentPrediction [26] is a diachronic multilingual dataset of 85K cases from the Federal Supreme Court of Switzerland (FSCS), where 50k cases in German, 31k in French, and 4K in Italian.…”

Section: Plea Judgment Prediction Datasetsmentioning

confidence: 99%

“…The availability of challenging benchmark datasets plays a crucial role in spurring innovation in LJP [22]. In the past few years, we have witnessed an explosion of public LJP benchmark datasets, such as CAIL2018 [23,24], ECHR-CASES [23,25], SwissJudgmentPrediction [26], JUSTICE [27,28]. These datasets inspired a large number of LJP models, such as TopJudge [29], MLCP-NLN [30], MPBFN-WCA [31], and LADAN [32].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges

Cui¹,

Shen²,

Nie³

et al. 2022

Preprint

View full text Add to dashboard Cite

Legal judgment prediction (LJP) applies Natural Language Processing (NLP) techniques to predict judgment results based on fact descriptions automatically. Recently, large-scale public datasets and advances in NLP research have led to increasing interest in LJP. Despite a clear gap between machine and human performance, impressive results have been achieved in various benchmark datasets. In this paper, to address the current lack of comprehensive survey of existing LJP tasks, datasets, models and evaluations, (1) we analyze 31 LJP datasets in 6 languages, present their construction process and define a classification method of LJP with 3 different attributes;(2) we summarize 14 evaluation metrics under four categories for different outputs of LJP tasks; (3) we review 12 legal-domain pretrained models in 3 languages and highlight 3 major research directions for LJP; (4) we show the state-of-art results for 8 representative datasets from different court cases and discuss the open challenges. This paper can provide up-to-date and comprehensive reviews to help readers understand the status of LJP. We hope to facilitate both NLP researchers and legal professionals for further joint efforts in this problem.

show abstract

Section: Datasetmentioning

confidence: 99%

Section: Charges Rmentioning

confidence: 99%

Section: Datasetsmentioning

confidence: 99%

Section: Plea Judgment Prediction Datasetsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges

Cui¹,

Shen²,

Nie³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges

Cui,

Shen,

Wen

2023

IEEE Access

View full text Add to dashboard Cite

Legal judgment prediction (LJP) applies Natural Language Processing (NLP) techniques to predict judgment results based on fact descriptions automatically. The present work addresses the growing interest in the application of NLP techniques to the task of LJP. Despite the current performance gap between machines and humans, promising results have been achieved in a variety of benchmark datasets, owing to recent advances in NLP research and the availability of large-scale public datasets. To provide a comprehensive survey of existing LJP tasks, datasets, models, and evaluations, this study presents the following contributions: (1) an analysis of 43 LJP datasets constructed in 9 different languages, together with a classification method of LJP based on three different attributes; (2) a summary of 16 evaluation metrics categorized into 4 different types to evaluate the performance of LJP models for different outputs;(3) a review of 8 legal-domain pretrained models in 4 languages, highlighting four major research directions for LJP; (4) state-of-the-art results for 11 representative datasets from different court cases and an in-depth discussion of the open challenges in this area. This study aims to provide a comprehensive review for NLP researchers and legal professionals to understand the advances in LJP over the past years, and to facilitate further joint efforts towards improving the performance of LJP models.

show abstract

Criminal Court Judgment Prediction System Built on Modified BERT Models

Latisha,

Favian,

Suhartono

2024

JAIT

View full text Add to dashboard Cite

The high crime rate in Indonesia that occurs from year to year causes a high number of cases that must be examined, tried, and decided through the courts as stipulated in Law No. 48 of 2009 concerning Judicial Power. Therefore, this study was conducted to build a system for predicting sentences resulting from criminal court decisions in the Republic of Indonesia which is expected to facilitate the implementation of jurisprudence. The prediction system was built by comparing 6 Bidirectional Encoder Representations from Transformers (BERT) models and a A Robustly Optimized BERT Pretraining Approach (RoBERTa) model on 3 different proposed architectures: BERT Base, Hierarchical BERT + Mean Pooling, and Hierarchical BERT + LSTM (Long Short-Term Memory). The compared models include indobert-base-p1, indobert-base-uncased, legalindobert-indonlu, legal-indobert-indolem, indobert-large-p1, indonesian-roberta-base. Those models are also compared with Support Vector Machine (SVM)+Term Frequency-Inverse Document Frequency (TF-IDF) as a baseline. The legal-indobert-indolem model with the Hierarchical BERT + Mean Pooling architecture succeeded in performing multiclass classification tasks into 14 classes with the highest F1score value of 79.8888%. Thus, the successfully created model can be further used in assisting jurisprudence as it has developed the ability to predict criminal court decisions based on similar previously documented cases.

show abstract

Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark

Cited by 4 publications

References 10 publications

A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges

A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges

A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges

Criminal Court Judgment Prediction System Built on Modified BERT Models

Contact Info

Product

Resources

About