Engineering experts critics for cooperative systems

Abstract

Knowledge collection systems often assume they are cooperating with an unbiased expert. They have few functions for checking and fixing the realism of the expertise transferred to the knowledge base, plan, document or other product of the interaction. The same problem arises when human knowledge engineers interview experts. The knowledge engineer may suffer from the same biases as the domain expert. Such biases remain in the knowledge base and cause difficulties for years to come.To prevent such difficulties, this paper introduces the reader to “critic engineering”, a methodology that is useful when it is necessary to doubt, trap and repair expert judgment during a knowledge collection process. With the use of this method, the human expert and knowledge-based critic form a cooperative system. Neither agent alone can complete the task as well as the two together.The methodology suggested here offers a number of extensions to traditional knowledge engineering techniques. Traditional knowledge engineering often answers the questions delineated in generic task (GT) theory, yet GT theory fails to provide four additional sets of questions that one must answer to engineer a knowledge base, plan, design or diagnosis when the expert is prone to error. This extended methodology is called “critic engineering”.

Knowledge maintenance: the state of the art

“…Silverman discussing representation independent methods for the engineering of expert critiquing systems (ECS) (Silverman, 1990(Silverman, , 1992b(Silverman, , 1992aSilverman and Wenig, 1993) which Silverman de®nes as follows:`F F F programs that ®rst cause their user to maximise the falsi®ability of their statements and then proceed to check to see if errors exist. A good critic program doubts and traps its user into revealing his or her errors.…”

“…Silverman has some experimental results (Silverman and Wenig, 1993) suggesting negative feedback (e.g. the use of a debiaser) is almost always unsuitable, if only used without positive feedback (in¯uencers).…”

“…age(patient)=old). The meaning of words must be maintained since the rest of the knowledge 110 base is a construction that connects these terms. Given a dependency network representing the inference ow in a system, the words would be the vertices (such a dependency network would be computed from knowledge of sentences and PSMs).…”

Introducing expert critiquing systems

“…Silverman's paper (Silverman, 1993) illuminates another important aspect of the expert critiquing approach, namely the issue of handling expert bias and human error in decision making situations. Thus, Silverman presents a methodology for critic engineering that is useful in supporting knowledge acquisition and representation, with a special emphasis on decreasing the degree of biased judgement in the elicitated knowledge.…”