Evaluating Expert Systems in a Financial Domain: A Study Demonstrating Issues in Case Selection, Validation Standards and Evaluator Bias

Abstract

This article describes and generalizes a validation study of four commercially available personal financial planning expert systems and the rationale for the research methodology used. Our evaluation of these systems adds to the understanding of verification and validation issues related to case selection, validation standards and evaluator bias. The article describes the systems, their domain and the empirical method-field tests using hypothetical cases-and relates that method to the literature. Comparing same-task systems combines multiple system perspectives and multiple models. Our methodology did efficiently and effectively identify conflicting terminology, omissions and system weaknesses but was inadequate for comparing the complex plan recommendations. The results re-emphasize the importance of continual knowledge base updating, formal system testing and the need for external evaluation. The results also show the value of comparing multiple, same-task systems.

A Hybrid‐Based Expert System for Personal Pension Planning in the UK

“…The nature of this difficulty is primarily due to the large number of valid paths through the system which result in acceptable solutions for any given scenario. It was, however, possible to follow an evaluation pattern similar to that suggested in Brown et al, (1993-from Grogono et al, 1991, and Preece, 1990. Unlike the problems faced by Brown et al,(p. 82), it was possible to have access to both the knowledge bases and the system output and it was therefore possible to develop tests of both the logical and empirical forms.…”

“…Solutions to this kind of problem are currently the subject of significant amounts of discussion (e.g. Berry and Hart, 1990;Hamilton et al, 1991;Brown et al, 1993). The nature of this difficulty is primarily due to the large number of valid paths through the system which result in acceptable solutions for any given scenario.…”