Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark

Abstract

In many jurisdictions, the excessive workload of courts leads to high delays. Suitable predictive AI models can assist legal professionals in their work, and thus enhance and speed up the process. So far, Legal Judgment Prediction (LJP) datasets have been released in English, French, and Chinese. We publicly release a multilingual (German, French, and Italian), diachronic (2000-2020) corpus of 85K cases from the Federal Supreme Court of Switzerland (FSCS). We evaluate state-of-the-art BERT-based methods including two variants of BERT that overcome the BERT input (text) length limitation (up to 512 tokens). Hierarchical BERT has the best performance (approx. 68-70% Macro-F1-Score in German and French). Furthermore, we study how several factors (canton of origin, year of publication, text length, legal area) affect performance. We release both the benchmark dataset and our code to accelerate future research and ensure reproducibility.

FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing

“…ECtHR The European Court of Human Rights (ECtHR) hears allegations that a state has breached human rights provisions of the European Convention of Human Rights (ECHR). We use the dataset of Chalkidis et al (2021), which contains 11K cases from ECtHR's public database. Each case is mapped to articles of the ECHR that were violated (if any).…”

“…The court often focus only on small parts of previous decision, where they discuss possible wrong reasoning by the lower court. The Swiss-Judgment-Predict dataset (Niklaus et al, 2021) contains more than 85K decisions from the FSCS written in one of three languages (50K German, 31K French, 4K Italian) from the years 2000 to 2020. The dataset provides labels for a simplified binary (approval, dismissal) classification task.…”

“…We standardize and put together four datasets: ECtHR (Chalkidis et al, 2021), SCOTUS (Spaeth et al, 2020), FSCS (Niklaus et al, 2021), and CAIL (Xiao et al, 2018;Wang et al, 2021b) that are already publicly available under CC-BY-(NC-)SA-4.0 licenses. We release the compiled version of the dataset under a CC-BY-NC-SA-4.0 license to favor academic research, and forbid to the best of our ability potential commercial dual use.…”

LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

“…The same or similar task has also been studied with court cases in many other jurisdictions including France (S ¸ulea et al, 2017), Philippines (Virtucio et al, 2018), Turkey (Mumcuoglu et al, 2021), Thailand (Kowsrihawat et al, 2018), United Kingdom (Strickson and De La Iglesia, 2020), Germany (Urchs et al, 2021), and Switzerland (Niklaus et al, 2021). Apart from predicting court decisions, there is also work aiming to interpret (explain) the decisions of particular courts (Ye et al, 2018;Chalkidis et al, 2021c;Branting et al, 2021).…”

Predicting Brazilian Court Decisions

“…In the context of the Supreme Court of Turkey, Mumcuoglu et al (2021) approach ( Mumcuoglu et al, 2021 ) has an accuracy of 93.2% and an F1-score of 0.87 for tax cases. In Niklaus, Chalkidis & Stürmer (2021) , the authors evaluate state-of-the-art BERT-based methods by using a multilingual corpus in German, French, and Italian with ∼85,000 records. The best results achieved an F1-macro score of 70% for German and French languages.…”