Laddering: technique and tool use in knowledge acquisition

The experimental evaluation of knowledge acquisition techniques and methods: history, problems and new directions

“…Some of this work was done under the remit of an Alvey project (Burton, Shadbolt, Hedgecock & Rugg, 1987;Schweickert, Burton, Taylor, Corlett, Shadbolt & Hedgecock, 1987;Burton, Shadbolt, Rugg & Hedgecock, 1988;Burton, Shadbolt, Rugg & Hedgecock, 1990b), and some under the ESPRIT project ACKnowledge (Burton et al, 1990a;Rugg & Shadbolt, 1991;Rugg, Corbridge, Major, Burton & Shadbolt, 1992;Corbridge, Rugg, Burton & Shadbolt, 1993;Corbridge, Rugg, Major, Shadbolt & Burton, 1994). See also Shadbolt and Burton (1990).…”

“…Some domains were real-world/commercial, such as the domain of lighting for industrial inspection (Schweickert et al, 1987), or the identi"cation of corrosion of metals (Corbridge, Rugg, Burton & Shadbolt, 1993). Other domains were real-world/academic/scienti"c, such as the identi"cation of #int artifacts from Stone Age tool production (Burton et al, 1990b) or the diagnosis of acute abdominal medical conditions (Corbridge et al, 1994). Others were &&toy'' domains, such as the identi"cation of fruit (Rugg et al, 1992), or methods of transport (Burton et al, 1990a).…”

“…There were a number of interesting features, such as the relatively poor performance of protocol analysis (Burton et al, 1988(Burton et al, , 1990b, the good performance of so-called contrived techniques against natural techniques ; see Shadbolt & Burton (1990) for a de"nition of the distinction], and the failure to show that software implementations of techniques outperformed pen-and-paper versions (Rugg et al, 1992;Corbridge et al, 1994), but these features are discussed more fully in the review paper , and the discussion sections of the various experimental reports. Our intention here is to try to evaluate the evaluations, to see how well the experimental designs coped with the inherent problems of software evaluation.…”

“…So, for example, Burton et al (1988) took 32 "nal year undergraduate geographers, and experiment 2 of Corbridge et al (1994) took 32 "nal year medical students. The advantages of this sort of experimental method, of course, are the greater numbers can be mustered, and they can be assessed against a gold standard provided by their teachers.…”

“…One approach was to make the attempt to use genuine experts, or near-experts, at postgraduate level or above. For instance, expert metallurgists were used for experiment 1 of Corbridge et al (1994), and expert archaeologists for experiments 1 and 2 of Burton et al (1990b). The problem, as expected, is bringing together and coordinating large enough numbers of such experts to provide statistical signi"cance: Corbridge et al used eight experts in corrosion, while in the Burton et al experiment 1, although 16 archaeologists were used, they were divided across two domains, with eight experts on #ints and eight experts on pottery.…”