Acknowledging insufficiency in the evaluation of legal knowledge-based systems

Abstract

This paper considers the need for evaluation of knowledge-based systems in general and legal knowledge-based systems in particular. Some special features of legal knowledge-based systems pertinent to their evaluation are presented. The expected benefits of such evaluations are discussed. and some of the difficulties likely to be encountered are outlined.The proceedings of four International Conferences on Artificial Intelligence and Law are analysed to determine the rate of reporting evaluations in non-theoretical papers. These papers had a low rate of consideration of evaluation issues reflecting common practice in research biased development environments. These results confirm that more attention to evaluation is needed in the legal knowledge based systems domain. This paper foreshadows the development of an evaluation methodology tailored specifically for legal knowledge-based systems. Evaluation strategies beyond verification and validation are drawn upon, both from the international ISO/IEC 14598 and 9126 standards and also from previous work on evaluation models for knowledge-based systems.

A history of AI and Law in 50 papers: 25 years of the international conference on AI and Law

“…It does, however, raise the question: why is there so little work like this in ICAIL? The discussion of Hall and Zeleznikow [127] in 9.4 will draw further attention to the general lack of evaluation in ICAIL papers. There is some empirical work, and some formalisation, but very few results in the form of theorems.…”

“…9.4 Jean Hall and John Zeleznikow. Acknowledging insufficiency in the evaluation of legal knowledge-based systems: Strategies towards a broad based evaluation model [127].…”

“…To the best of our knowledge, [127] represents the first self-reflexive work from and about the ICAIL conference. The point the authors make is that there is a deficiency of reported evaluation in the field of legal knowledge-based systems.…”

“…The operative word in [127] is evaluation. In varying degrees, this work covers three related areas corresponding to three separate levels of granularity: "conventional" software systems, knowledge-based systems (KBS), and legal knowledge-based systems (LKBS).…”

Towards a New MODDE of Improving Legal Decision Support Design

“…Evaluation of t can be achieved through examining user satisfaction in its many permutations (Hall & Zeleznikow, 2001). Intentional t is achieved through an appropriate software engineering process involving thorough investigation and understanding of the domain, generation of effective software requirements, and creation of a suitable software solution-all within a professional approach that permits responsiveness to changing circumstances and is primarily user-focused (for some interesting viewpoints, see Damodaran, 1996;Kaulio & Karlsson, 1998;Tata, 1998).…”