Rapid advances in data analysis techniques—particularly for predictive algorithms—have opened the door for radically new perspectives on legal practice and access to justice. Several firms in North America, Asia, and Europe have set out to use machine-learning techniques to generate legal predictions, raising concerns regarding ethics, reliability and limits on prediction accuracy, and potential impact on case law development. To explore these opportunities and challenges, we consider in depth one of the most litigated issues in Canada: wrongful termination disputes and, more specifically, the question of reasonable notice determination. Beyond the thorough analysis of this question, this paper is also intended to act as a road map for non-technicians (and especially lawyers) on the application of artificial intelligence (AI) methods, illustrating both their potential benefits and limitations in other areas of dispute resolution. To achieve these results, we first created a large dataset by annotating historic cases related to employment termination. This dataset proved useful for assessing the predictability of reasonable notice of termination, that is, the accuracy and precision of AI predictions. In particular, it helped identify the degree of inconsistency in notice period cases, incidentally exposing the limitations of legal predictions. We then developed predictive algorithms to estimate notice periods based on details of the employment period and investigated their accuracy and performance. Moreover, we thoroughly analyzed these algorithms to better understand the judicial process, and in particular to quantify the weight and influence of case-specific features in the determination of reasonable notice. Finally, we closely analyzed cases that were poorly predicted by the algorithms to understand the judicial decision-making process and identify inconsistencies—a strategy that will ultimately yield a deeper practical understanding of case law. This project will open the door to the development of an access-to-justice project and will provide users with an open-access platform for employment legal help (www.MyOpenCourt.org).

In the past decade deep learning models have achieved impressive performance on a wide range of tasks. However, they still face challenges in many high-stakes problems. In this paper we study Legal Judgment Prediction (LJP), which is an important high-stakes task utilizing fact descriptions obtained from court cases to make final judgements. We investigate the state-of-the-art of the LJP task by leveraging the most recent deep learning models, longformer, and demonstrate that we obtain the state-of-the-art performance, even with a limited amount of training data, benefiting from the advantage of pretraining and the long-sequence modeling capability of longformer. However, our analyses suggest that the improvement is due to the model's fitting to spurious correlations, in which the model makes correct decisions based on information irrelevant to the task itself. We advocate that caution should be seriously exercised when explaining the obtained results. The second challenge in many high-stakes problems is interpretability required for models. The final predictions made by deep learning models are useful only if the evidences that support the models' decisions are consistent with those used by subject-matter experts. We demonstrate that by using post-hoc interpretation, the conventional method XGBoost is actually capable of providing explainable results with a performance comparable to the longformer model, while not being subject to the spurious correlation issue. We hope our work contributes to the line of research on understanding the advantages and limitations of deep learning for high-stakes problems.

No abstract