Analysis and Multilabel Classification of Quebec Court Decisions in the Domain of Housing Law

Abstract

The Régie du Logement du Québec (RDL) is a tribunal with exclusive jurisdiction in matters regarding rental leases. Within the framework of the ACT (Autonomy Through Cyberjustice Technologies) project, we processed an original collection of court decisions in French and performed a thorough analysis to reveal biases that may influence prediction experiments. We studied a multilabel classification task that consists in predicting the types of verdict in order to illustrate the importance of prior data analysis. Our best model, based on the FlauBERT language model, achieves F1 score micro averages of 93.7% and 84.9% in Landlord v. Tenant and Tenant v. Landlord cases respectively. However, with the support of our in-depth analysis, we emphasize that these results should be kept in perspective and that some metrics may not be suitable for evaluating systems in sensitive domains such as housing law.

From One to Many: Identifying Issues in CJEU Jurisprudence

“…Judgments concern distinct legal question and knowing what those questions are and which judgments address which question is arguably the most important knowledge that lawyers have, but they are also important to scholars seeking to understand how courts behave in different areas of law. Automated text analysis allows us to predict topic labels from text (see, e.g., Aletras et al 2016;Ashley and Bruninghaus 2009;Salaün et al 2020), typically using latent Dirichlet allocation (LDA) topic modeling, which summarizes a corpus assuming an unknown structure of topics reflected in the individual documents of the corpus. Previous studies have used LDA topic modeling to identify topics in different bodies of case law (Carter et al 2016;Lauderdale and Clark 2014;Panagis et al 2016;Soh et al 2019;Trappey et al 2020;Venkatesh and Raghuveer 2013).…”

Pretrained Language Models v. Court Ruling Predictions: A Case Study on a Small Dataset of French Court of Appeal Rulings

“…As regards legal NLP applied to French, Şulea et al (2017) and Sulea et al (2017) evaluated SVM classifiers on the prediction of the area of a case, as well as its ruling, on decisions from the French Supreme Court (Cour de Cassation). Salaün et al (2020) used pretrained language models such as Flaubert and Camembert (Martin et al, 2020) to predict the outcome of cases about conflicts between a tenant and a landlord in Québec. They observed that pretrained language models outperformed linear classifiers.…”

Rethinking the field of automatic prediction of court decisions

“…3 For description of earlier approaches in automatic prediction of court decision with and without using machine learning we refer to Ashley and Brüninghaus (2009) restricted set of courts. In this paper, we surveyed publications that use machine learning approaches and focus on case-law of the US Supreme Court (Sharma et al 2015;Katz et al 2017;Kaufman et al 2019), the French court of Cassation (Şulea et al 2017b;Sulea et al 2017a), the European Court of Human Rights (Aletras et al 2016;Liu and Chen 2017;Chalkidis et al 2019;Kaur and Bozic 2019;O'Sullivan and Beel 2019;Visentin et al 2019;Chalkidis et al 2020;Condevaux 2020;Medvedeva et al 2020a, b;Quemy and Wrembel 2020;Medvedeva et al 2021), Brazilian courts (Bertalan and Ruiz 2020;Lage-Freitas et al 2019), Indian courts (Bhilare et al 2019;Shaikh et al 2020;Malik et al 2021), UK courts (Strickson and De La Iglesia 2020), German courts (Waltl et al 2017), the Quebec Rental Tribunal (Salaün et al 2020) (Canada), the Philippine Supreme Court (Virtucio et al 2018), the Thai Supreme Court (Kowsrihawat et al 2018) and the Turkish Constitutional Court (Sert et al 2021). Many of these papers achieve a relatively high performance on their specific task using various machine learning techniques.…”

“…Finally, Salaün et al (2020) essentially combined the two types of predictors, by not only extracting a number of characteristics from the cases of Rental Tribunal of Quebec (including the court location, judge, types of parties, et cetera), but also using the raw text of the facts (as well as the complete text excluding the verdict), achieving a performance of at most 85% with a French BERT model, FlauBERT.…”