Comparative evaluation of expert system testing methods

Abstract

Effective testing (validation, verification and evaluation) for expert systems is becoming important. In this paper we do a comparative evaluation of blackbox, white-boz, consistency and completeness testing methods based on the criteria of eflectiueness, robustness and cost. Testing methods are evaluated using "life-cycle mutation testing" on a VLSI manufacturing diagnostic expert system.

Evaluation of verification tools for knowledge-based systems

“…This confirms the previous studies which have used manual or automated versions of these kinds of verification. In particular, our results agree with the Concordia and SAIC studies of consistency and completeness checking (Preece & Shinghal, 1992;Miller, Hayes & Mirsky, 1993), and the SRI and Minnesota studies of testing (Rushby & Crow, 1990;Kirani et al, 1992).…”

“…In view of this, our fault classes are based on the implementation. They are similar to those of the SAIC study (Miller, Hayes & Mirsky, 1993), and a subset of those in the Minnesota study (Kirani et al, 1992) (which defined fault classes for requirements and design documents also). We accept that faults may be introduced prior to implementation (e.g.…”

“…The relative efficacy of the consistency and completeness checking methods appear to contradict directly the conclusions of the Minnesota study (Kirani et al, 1992). Of course, neither the tools, the application nor the method are the same and this can explain the differences.…”

“…Of course, neither the tools, the application nor the method are the same and this can explain the differences. For example the application used in Kirani et al (1992), MAPS, is very small (forward chaining, propositional logic, with only 41 rules) compared to the application we used, GIBUS (object-oriented, forward and backward chaining, first-order logic, with 207 rules). The use of a small application can be justified by the fact that evaluation and analysis are easier.…”

“…¹he Minnesota study by Kirani, Zualkernan and Tsai (1992) at the University of Minnesota, USA, report on the application of several testing techniques to a sample KBS in the domain of VLSI manufacturing. With the exception of a simple consistency checking tool, all of the methods used were manual testing techniques.…”