Comparing knowledge elicitation techniques: a case study

Abstract

Three knowledge elicitation techniques were used to extract knowledge bases from experts on lighting for industrial inspection tasks. The techniques were: I a structured interview; II 'twenty questions'-imputing rules from information requests; and IIIa card sort. The first two techniques generate protocols, and in these cases two knowledge engineers independently extracted production rules from the protocols. In the third technique rules were derived from the classification of lighting solutions. The first two techniques led to about the same number of rules, the card sort to less. There was a slightly higher percentage of agreement between the rules extracted separately by the knowledge engineers for the normal interview than for the 'twenty questions' technique. Disagreements between the knowledge engineers were resolved by discussion, and an expert system to select special lights for inspection tasks was implemented. Of the rules finally agreed on, the percentage which could be implemented in the expert system was less for the twenty questions technique than for the others. The agreed rules were mailed to the expert, who indicated whether she agreed, disagreed, or wanted to make a modification. This secondary stage of elicitation revealed no evident difference between the three techniques in terms of the proportion of rules validated by the expert.

Eliciting Constructability Knowledge for BIM-enabled Automated, Rule-based Constructability Review: A Case Study of Formwork

“…Comparing the operation and outcomes of both elicitation techniques, six dimensions were used to examine the efficacy to acquire the knowledge regarding formwork selection for automated constructability review of a given design (Table 1; Schweickert et al, 1987;Hoffman et al, 1995). The advantages and limitations of each technique have been revealed in Table 1.…”

“…The distinction between the four different types of knowledge implies that various elicitation techniques are needed for soliciting the proper type of knowledge (Diaper, 1989). To capture the strengths and weakness of each technique, criteria used can be related to its operation -brevity, productivity, and accessibility of knowledge source, and its outcome -results validity, breadth of knowledge, and differential access (Schweickert et al, 1987;Hoffman et al, 1995). Regardless, interview and document analysis were found to be most frequently used (Cullen and Bryman, 1988).…”

The experimental evaluation of knowledge acquisition techniques and methods: history, problems and new directions

“…Attention was paid to getting a range of domains each of which might set di!erent problems for KA techniques. Some domains were real-world/commercial, such as the domain of lighting for industrial inspection (Schweickert et al, 1987), or the identi"cation of corrosion of metals (Corbridge, Rugg, Burton & Shadbolt, 1993). Other domains were real-world/academic/scienti"c, such as the identi"cation of #int artifacts from Stone Age tool production (Burton et al, 1990b) or the diagnosis of acute abdominal medical conditions (Corbridge et al, 1994).…”

“…A variation of this approach was to use a single genuine expert (Schweickert et al, 1987;Rugg & Shadbolt, 1991). This allowed a deeper interrogation to take place, but of course made generalization problematic.…”

A Theory of Inference Derivation for Qualitative Data: Development and Test with Application to Criminal and Terrorist Detection

“…They conclude that whilst protocol analysis performs poorly in their elicitation tests, the more 'contrived' techniques such as laddered grids and card sorts are as valuable as the more traditionally favoured interviews. This finding was contrary to the views of the experts themselves who believed the 209 !~ techniques to be the least successful (Schweickert et al 1987). They conclude that 'It appears that an expert's view of what constitutes a good KE session is not predictive of its real efficacy'.…”