Memory recall in a process control system: a measure of expertise and display effectiveness

Abstract

Previous research has shown that memory-recall performance is correlated with domain expertise. In this study, a process control system was selected as a vehicle for conducting research on memory recall. The primary purposes of the present work were to determine if the classic expertise effects originally obtained in chess generalize to this novel domain and to evaluate the validity of memory recall as a measure of display effectiveness. Experts and novices viewed dynamic event sequences showing the behavior of a thermal-hydraulic system with two different displays, one that only contained information about the physical components in the system (P) and another that also contained information about higher order functional variables (P +F). There were three types of trials: normal, where the system was operating correctly; fault, where a single fault was introduced; and random, where the system's behavior did not obey physical laws. On each trial, subjects were asked to recall the final state of the system and to diagnose the system state. The P +F display resulted in superior diagnosis performance compared with the P display. With regard to memory, there was some evidence of an interaction between trial type and expertise, with experts outperforming novices but primarily on meaningful trials. In addition, memory for the subset of variables most critical to diagnosis was better with the P +F display than with the P display, thereby indicating that memory recall can be a sensitive measure of display effectiveness. The results also clarify a theoretical problem that has existed for some time in the literature, namely, the conditions under which expertise advantages are to be expected in memory-recall tasks. Collectively, these findings point to the potential benefits of adopting an applied context as a test bed for basic research issues.Research on the use of memory recall as a measure of domain expertise originated at least 45 years ago with the seminal work of de Groot (1946Groot ( /1965 on problem solving in chess. In that experiment, four chess players of various levels of expertise were asked to reconstruct meaningful board positions after having been exposed to them for only a few seconds. De Groot discovered that the Master-and Grandmaster-level players were able to perform this task with near-perfect accuracy, whereas the performance of the lesser playe~s was not nearly a~im-pressive. In a subsequent expenment, Chase and Sunon

Cognitive flexibility and adaptability to environmental changes in dynamic complex problem-solving tasks

“…In most research, an expert is defined explicitly or implicitly as someone who has spent 10 years of extended, deliberate practice Smith 1991, Ericsson andLehmann 1996) in one particular area. However, the experts used in other research, related to laboratory studies on process control like the present one (see Vicente 1992, used also in Vicente and Wang 1998), were engineering graduate students whose only direct experience of the complex system was restricted to an hour-long general introduction given to all participants before the start of the experiment.…”

Cognitive Ability, Expertise, and Age Differences in Following Air-Traffic Control Instructions.

“…More specifically, investigators need to use tasks in which the stimuli are familiar and organized and the responses are compatible with the goals of the relevant performance domain (Vicente, 1992;Vicente & Wang, 1998). However, even when investigators designed their tasks to be domain relevant, studies of skilled memory have still failed to obtain significant Expertise ϫ Age interactions (e.g., Meinz & Salthouse, 1998;Morrow et al, 2001).…”

Mental Models for Effective Training: Comparing Expert and Novice Maintainers' Mental Models

“…The seminal example comes from Chase and Simon (1973) where expert chess players reconstructed board positions (and were generally successful) not piece by piece but in related groups, whereas novice chess players reconstructed board positions piece by piece (and were no more accurate than would be predicted by memory span limits). Another example comes from Vicente (1992), on a task in which participants viewed process-control simulations and estimated final processcontrol variable values, where expert mechanical engineering graduate students estimated variable values better than novice non-engineering graduate students when inputs to the system were meaningful, but not when random. These kinds of tasks inform the investigator of models that the participants might be following to produce the items recalled or the estimates made.…”

“…First, in this research only textual materials, that is words or short phrases, were used as stimulus items, a limit imposed by the online tool that may be removed in future versions of the tool or by use of another categorization tool. Diagrams in particular, but other stimulus forms (e.g., images of TMDE or of TMDE outputs) too, have been used in past research (e.g., Chi et al, 1981;Vicente, 1992) to better understand participants' mental models particularly of complex equipment such as is appropriate to Ordnance maintenance technicians.…”