Verifying, validating, and measuring the performance of expert systems

Suen, Ching Y.; Grogono, Peter; Shinghal, Raijan; Coallier, François

doi:10.1016/0957-4174(90)90019-q

Cited by 41 publications

(11 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is no different from the situation reported in the general KBS community [11,25,50]. A number of reasons could be contributory factors.…”

Section: Discussion Of Survey Resultscontrasting

confidence: 58%

“…insufficient available practical experience and critical appraisal of such experience [11]. Suen concurs, suggesting that the need for evaluation was not previously recognized due to the small number of available systems [50]. Today's LKBS developer is unlikely to have received training in a KBS development environment where evaluation is prevalent or have seen many (if indeed any) systems that have been extensively evaluated.…”

Section: Discussion Of Survey Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Acknowledging insufficiency in the evaluation of legal knowledge-based systems

Hall

Zeleznikow

2001

Proceedings of the 8th International Conference on Artificial Intelligence and Law

View full text Add to dashboard Cite

This paper considers the need for evaluation of knowledge-based systems in general and legal knowledge-based systems in particular. Some special features of legal knowledge-based systems pertinent to their evaluation are presented. The expected benefits of such evaluations are discussed. and some of the difficulties likely to be encountered are outlined.The proceedings of four International Conferences on Artificial Intelligence and Law are analysed to determine the rate of reporting evaluations in non-theoretical papers. These papers had a low rate of consideration of evaluation issues reflecting common practice in research biased development environments. These results confirm that more attention to evaluation is needed in the legal knowledge based systems domain. This paper foreshadows the development of an evaluation methodology tailored specifically for legal knowledge-based systems. Evaluation strategies beyond verification and validation are drawn upon, both from the international ISO/IEC 14598 and 9126 standards and also from previous work on evaluation models for knowledge-based systems.

show abstract

“…This is no different from the situation reported in the general KBS community [11,25,50]. A number of reasons could be contributory factors.…”

Section: Discussion Of Survey Resultscontrasting

confidence: 58%

Section: Discussion Of Survey Resultsmentioning

confidence: 99%

Acknowledging insufficiency in the evaluation of legal knowledge-based systems

Hall

Zeleznikow

2001

Proceedings of the 8th International Conference on Artificial Intelligence and Law

View full text Add to dashboard Cite

show abstract

“…Consequently, considerable work has been done in this area since expert systems were first realized as practical tools, rather than A1 experiments (Gaschnig et a1 1983). Over the last seven years, the expert system evaluation literature has rapidly expanded, and many alternative methods have been proposed for verification, validation and user acceptance testing (for a recent survey, see Suen et al 1990). At the same time, there has been considerable interest in formalizing expert system development methodology, to bring knowledge-based systems into line with the structured methods of conventional software engineering (Guida & Tasso 1989).…”

Section: Introduction and Overviewmentioning

confidence: 99%

Towards a methodology for evaluating expert systems

Preece

1990

Expert Systems

View full text Add to dashboard Cite

Expert system evaluation comprises verification, validation and user acceptance testing. The nature of expert systems requires that they be elvaluated carefully, and that detailed methodologies for their development he devised. This paper attempts to give practical guidance for the e\,aluation phase of' expert system de\-elopment. Empirical and logical e~alrrution methods are surveyed, and both types of method are applied in an evaluation case study. This case study provides insights as to the relative effectiveness of the methods. and demonstrates that empirical and logical testing are complementary to each other. The paper concludes that any eiwluation methodology should incorporate both types of method, and presents prescwptions and an orrtline for such a methodology, which combines the methods for maximum effectiveness.

show abstract

“…Several authors (Suen et al, 1990;Lehner, 1989;O'Keefe et al, 1987) recommend that test case results be statistically evaluated. Statistical measurement for validation assumes the expert system results can be easily encoded.…”

Section: Discussionmentioning

confidence: 99%

Evaluating Expert Systems in a Financial Domain: A Study Demonstrating Issues in Case Selection, Validation Standards and Evaluator Bias

Brown

Nielson

Phillips

1993

Intelligent Systems in Accounting, Finance and Management

View full text Add to dashboard Cite

This article describes and generalizes a validation study of four commercially available personal financial planning expert systems and the rationale for the research methodology used. Our evaluation of these systems adds to the understanding of verification and validation issues related to case selection, validation standards and evaluator bias. The article describes the systems, their domain and the empirical method-field tests using hypothetical cases-and relates that method to the literature. Comparing same-task systems combines multiple system perspectives and multiple models. Our methodology did efficiently and effectively identify conflicting terminology, omissions and system weaknesses but was inadequate for comparing the complex plan recommendations. The results re-emphasize the importance of continual knowledge base updating, formal system testing and the need for external evaluation. The results also show the value of comparing multiple, same-task systems.

show abstract

Verifying, validating, and measuring the performance of expert systems

Cited by 41 publications

References 9 publications

Acknowledging insufficiency in the evaluation of legal knowledge-based systems

Acknowledging insufficiency in the evaluation of legal knowledge-based systems

Towards a methodology for evaluating expert systems

Evaluating Expert Systems in a Financial Domain: A Study Demonstrating Issues in Case Selection, Validation Standards and Evaluator Bias

Contact Info

Product

Resources

About