Software Visualization can be defined as the use of graphical and textual formalisms to describe the execution of computer programs. A large amount of Software Visualization technology has been developed to support computer science education, using a range of interface techniques. Far less effort has been devoted to evaluating the technology. As a result, it is unclear how effective Software Visualization tools are, either for students or professional programmers. Even more worrying, it is doubtful whether lessons are being learnt in successive designs of Software Visualization tools, or whether the application of new technologies (e.g. 3D animation and the intemet) has become the primary goal, rather than the true goal of making computer programs easier to understand. To counter this problem the study reported here used protocol analysis to develop a fine-grained account of user behaviour, identifying (i) information access from the display, (ii) the use of comprehension strategies, and (iii) misunderstandings of the visualization and execution. The results were able to motivate future deigns which in turn could be compared and improved. The approach is compared to other evaluation techniques which aim to inform design. Finally, the generalizability of the approach is considered.
INTRODUCTIONThis paper presents a framework for the principled evaluation and design of Software Visualization (SV) technology. SV is the process of using techniques such as typography, graphic design, animation and cinematography to provide representations of a program and its execution. Though a great deal of research effort is being devoted to SV development, SVs are still not widely used in practice, either by professional programmers or within computer programming education. Price, Small and Baecker (1993) argue an important reason why the technology has not been readily taken up is because its benefits have not been clearly demonstrated. Allied to this, is the uncertainty as to whether SV technology is improving, by learning lessons from earlier systems and using these lessons to motivate designs of the future. The SV field is certainly progressing technologically, though whether it is progressing from a cognitive or educational perspective is unclear.There have been a few empirical evaluations of SV technology. These have generally been coarsegrained, measuring how quickly or successfully certain tasks can be carried out with various SVs. These studies can provide estimates as to which SV "is best" for certain tasks or users but tell us little about "why". This paper reports on the development and application of a new methodology.The methodology uses protocol analysis (Ericsson and Simon, 1984) to gain cognitive evidence as to how SVs are used and understood. Previous research in the psychology of programming can be used to determine the types of cognitive evidence we can expect to identify within the protocols of subjects using a SV. SVs are intended to support the programmer in the tasks of program comprehension and de...