JEC-QA: A Legal-Domain Question Answering Dataset

Abstract

We present JEC-QA, the largest question answering dataset in the legal domain, collected from the National Judicial Examination of China. The examination is a comprehensive evaluation of professional skills for legal practitioners. College students are required to pass the examination to be certified as a lawyer or a judge. The dataset is challenging for existing question answering methods, because both retrieving relevant materials and answering questions require the ability of logic reasoning. Due to the high demand of multiple reasoning abilities to answer legal questions, the state-of-the-art models can only achieve about 28% accuracy on JEC-QA, while skilled humans and unskilled humans can reach 81% and 64% accuracy respectively, which indicates a huge gap between humans and machines on this task. We will release JEC-QA and our baselines to help improve the reasoning ability of machine comprehension models. You can access the dataset from http://jecqa.thunlp.org/.

How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence

“…Besides, COLIEE (Kano et al, 2018) contains about 500 yes/no questions. Moreover, the bar exam is a professional qualification examination for lawyers, so bar exam datasets (Fawei et al, 2016;Zhong et al, 2019a) may be quite hard as they require professional legal knowledge and skills.…”

“…We select JEC-QA (Zhong et al, 2019a) as the dataset of the experiments, as it is the largest dataset collected from the bar exam, which guarantees its difficulty. JEC-QA contains 28, 641 multiple-choice and multiple-answer questions, together with 79, 433 relevant articles to help to answer the questions.…”

“…For more qualified LQA methods, there are several significant difficulties to overcome: (1) Legal multi-hop reasoning. As Zhong et al (2019a) state, existing models can perform inference but not multi-hop reasoning. However, legal cases are very complicated, which cannot be handled by singlestep reasoning.…”

“…An example of LQA fromZhong et al (2019a). In this example, direct evidence and extra evidence are both required for answering the question.…”

Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text

“…Though several pre-trained language models have been introduced for domain-specific MRC, BERT based models are not as consistently dominant as they are in open field MRC tasks (Zhong et al, 2020;Yue et al, 2020). Another challenge is that medical questions are often more difficult; no labeled paragraph contains the answer to a given question.…”