Pragmatic Testing Methods for Expert Systems

Abstract

Effective testing of expert systems is an important but difficult task. In this paper we discuss why testing of expert systems is hard. For testing expert systems, testing techniques from conventional software engineering are adapted as a solution to these problems. We discuss the feasibility and application of black-box, white-box, and life-cycle testing techniques to expert systems. Black-box techniques include random, input partition and output partition testing. White-box techniques include path based partition, cause-effect graph, dynamic-flow, data-flow, and ablation testing. The use of these methods is demonstrated by applying the techniques to an industrial expert system. The results obtained are 1) random testing is feasible for expert systems with a small input space, 2) effectiveness of partition testing depends on the partition criteria, however, partition testing points out failure-prone partitions, 3) cause-effect testing is suitable for large expert systems if it is well decomposed, 4) dynamic-flow testing depends on the number of paths in solution and design specifications, 5) data-flow testing is feasible only if define-use pairs are obtained automatically, 6) ablation testing helps in pointing out discrepancies between specification and implementation. The overall result is that the expert systems can be treated as any other software and all the conventional software engineering testing techniques can be used to test expert systems.

Comparative evaluation of expert system testing methods

“…Generating expected outputs is often difficult because testing criteria can be unclear, output spaces for selection of test cases can be huge, lack of oracles (e.g. experts) or number of paths through a program can be large (for a detailed explanation see [19]). Table 6 Cause-effect Dynamic-flow Cost of loss is the cost incurred if a testing method fails to identify a fault.…”

“…We use a technique called "Life-cycle mutation testing" (LCMT) for a comparative evaluation of testing methods [24] on an expert system. We demonstrate the use of this technique by evaluating pragmatic testing techniques (blackbox and white-box testing techniques [19]), completeness and consistency testing techniques [12, 211 for a VLSI manufacturing diagnostic expert system called MAPS [7].…”

Verification and validation of knowledge-based systems

“…Cost of test case generation consists of cost associated with generation of test inputs and expected outputs. Generating expected outputs can be difficult because testing criteria can be unclear or due to lack of oracles [44]. Consistency and completeness methods have less cost of generation compared with dynamic testing methods (see Table 5) [22].…”