The efficacy of knowledge elicitation techniques: a comparison across domains and levels of expertise

The experimental evaluation of knowledge acquisition techniques and methods: history, problems and new directions

“…For instance, experiment 1 of Burton et al reported some di!erent patterns between the #int domain and the pottery domain (Burton et al, 1990b, p. 171). Furthermore*perhaps more worryingly*there was an important discrepancy between the results on feedback e!ects between Burton et al (1990b) and Corbridge et al (1993), the latter having a sample three times the size of the former. Corbridge et al found that giving the subjects feedback in the form of &&pseudo-production rules'' had no e!ect EXPERIMENTAL EVALUATION OF KA TECHNIQUES AND METHODS on the gain of the elicitation sessions, where Burton et al had found feedback e!ects on gain.…”

“…Some domains were real-world/commercial, such as the domain of lighting for industrial inspection (Schweickert et al, 1987), or the identi"cation of corrosion of metals (Corbridge, Rugg, Burton & Shadbolt, 1993). Other domains were real-world/academic/scienti"c, such as the identi"cation of #int artifacts from Stone Age tool production (Burton et al, 1990b) or the diagnosis of acute abdominal medical conditions (Corbridge et al, 1994). Others were &&toy'' domains, such as the identi"cation of fruit (Rugg et al, 1992), or methods of transport (Burton et al, 1990a).…”

“…There were a number of interesting features, such as the relatively poor performance of protocol analysis (Burton et al, 1988(Burton et al, , 1990b, the good performance of so-called contrived techniques against natural techniques ; see Shadbolt & Burton (1990) for a de"nition of the distinction], and the failure to show that software implementations of techniques outperformed pen-and-paper versions (Rugg et al, 1992;Corbridge et al, 1994), but these features are discussed more fully in the review paper , and the discussion sections of the various experimental reports. Our intention here is to try to evaluate the evaluations, to see how well the experimental designs coped with the inherent problems of software evaluation.…”

“…One approach was to make the attempt to use genuine experts, or near-experts, at postgraduate level or above. For instance, expert metallurgists were used for experiment 1 of Corbridge et al (1994), and expert archaeologists for experiments 1 and 2 of Burton et al (1990b). The problem, as expected, is bringing together and coordinating large enough numbers of such experts to provide statistical signi"cance: Corbridge et al used eight experts in corrosion, while in the Burton et al experiment 1, although 16 archaeologists were used, they were divided across two domains, with eight experts on #ints and eight experts on pottery.…”

“…For example, Burton et al experiment 1 seemed to con"rm the relative lack of e$ciency of protocol analysis in terms of resource consumption, despite the large amount of knowledge eventually captured (Burton et al, 1990b, p. 171; see also Burton et al, 1987Burton et al, , 1988. Furthermore, the experimental designs were created to facilitate detection of e!ects among small groups of subjects (Corbridge et al, 1994, pp.…”