Factoring Statutory Reasoning as Language Understanding Challenges

Holzenberger, Nils; Durme, Benjamin Van

doi:10.18653/v1/2021.acl-long.213

Cited by 11 publications

(11 citation statements)

References 27 publications

(22 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Servantez et al captured rules in contract text through graph-based extraction and converted them into code (Servantez et al, 2023). Similar to our work, Holzenberger and Van Durme introduced an approach to reasoning about tax statutes by decomposing the reasoning process (Holzenberger and Van Durme, 2021). This approach first extracts key arguments (entities, dates, dollar amounts) from the statute text and fact pattern using fine-tuned BERT models, before arriving at a final answer.…”

Section: Related Workmentioning

confidence: 96%

Computable Contracts by Extracting Obligation Logic Graphs

Servantez,

Lipka,

Siu

et al. 2023

Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law

View full text Add to dashboard Cite

Rule-based reasoning, a fundamental type of legal reasoning, enables us to draw conclusions by accurately applying a rule to a set of facts. We explore causal language models as rulebased reasoners, specifically with respect to compositional rules -rules consisting of multiple elements which form a complex logical expression. Reasoning about compositional rules is challenging because it requires multiple reasoning steps, and attending to the logical relationships between elements. We introduce a new prompting method, Chain of Logic, which elicits rule-based reasoning through decomposition (solving elements as independent threads of logic), and recomposition (recombining these sub-answers to resolve the underlying logical expression). This method was inspired by the IRAC (Issue, Rule, Application, Conclusion) framework, a sequential reasoning approach used by lawyers. We evaluate chain of logic across eight rule-based reasoning tasks involving three distinct compositional rules from the LegalBench benchmark and demonstrate it consistently outperforms other prompting methods, including chain of thought and self-ask, using open-source and commercial language models.

show abstract

Section: Related Workmentioning

confidence: 96%

Computable Contracts by Extracting Obligation Logic Graphs

Servantez,

Lipka,

Siu

et al. 2023

Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law

View full text Add to dashboard Cite

show abstract

“…Similarly, legal question answering (LQA) could offer affordable, expert-like assistance to the masses, thereby empowering marginalized parties when utilized for public welfare. However, existing research on LQA tends to exhibit a constrained scope, often concentrating on specialized legal domains, such as tax law (Holzenberger et al 2020) or privacy policies (Ravichander et al 2019), or limiting the responses to uninformative brief answers like yes/no replies (Rabelo et al 2022) or few-word spans (Duan et al 2019).…”

Section: Introductionmentioning

confidence: 99%

Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models

Louis,

Van Dijck,

Spanakis

2024

AAAI

View full text Add to dashboard Cite

Many individuals are likely to face a legal dispute at some point in their lives, but their lack of understanding of how to navigate these complex issues often renders them vulnerable. The advancement of natural language processing opens new avenues for bridging this legal literacy gap through the development of automated legal aid systems. However, existing legal question answering (LQA) approaches often suffer from a narrow scope, being either confined to specific legal domains or limited to brief, uninformative responses. In this work, we propose an end-to-end methodology designed to generate long-form answers to any statutory law questions, utilizing a "retrieve-then-read" pipeline. To support this approach, we introduce and release the Long-form Legal Question Answering (LLeQA) dataset, comprising 1,868 expert-annotated legal questions in the French language, complete with detailed answers rooted in pertinent legal provisions. Our experimental results demonstrate promising performance on automatic evaluation metrics, but a qualitative analysis uncovers areas for refinement. As one of the only comprehensive, expert-annotated long-form LQA dataset, LLeQA has the potential to not only accelerate research towards resolving a significant real-world issue, but also act as a rigorous benchmark for evaluating NLP models in specialized domains. We publicly release our code, data, and models.

show abstract

“…Article-aware classification has been explored on Chinese criminal case corpora (Wang et al, , 2019bYue et al, 2021;Chen et al, 2022). Similarly, Holzenberger et al 2020 has modeled statutory reasoning by classifying US tax law provisions concatenated with textual case descriptions. We build on this prior work in two ways.…”

Section: Introductionmentioning

confidence: 99%

Zero-shot Transfer of Article-aware Legal Outcome Classification for European Court of Human Rights Cases

T.y.s.s,

Ichim,

Grabmair

2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

In this paper, we cast Legal Judgment Prediction on European Court of Human Rights cases into an article-aware classification task, where the case outcome is classified from a combined input of case facts and convention articles. This configuration facilitates the model learning some legal reasoning ability in mapping article text to specific case fact text. It also provides an opportunity to evaluate the model's ability to generalize to zero-shot settings when asked to classify the case outcome with respect to articles not seen during training. We devise zero-shot experiments and apply domain adaptation methods based on domain discrimination and Wasserstein distance. Our results demonstrate that the article-aware architecture outperforms straightforward fact classification. We also find that domain adaptation methods improve zero-shot transfer performance, with article relatedness and encoder pre-training influencing the effect.

show abstract

Factoring Statutory Reasoning as Language Understanding Challenges

Cited by 11 publications

References 27 publications

Computable Contracts by Extracting Obligation Logic Graphs

Computable Contracts by Extracting Obligation Logic Graphs

Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models

Zero-shot Transfer of Article-aware Legal Outcome Classification for European Court of Human Rights Cases

Contact Info

Product

Resources

About