The challenge of information visualization evaluation

Abstract

As the field of information visualization matures, the tools and ideas described in our research publications are reaching users. The reports of usability studies and controlled experiments are helpful to understand the potential and limitations of our tools, but we need to consider other evaluation approaches that take into account the long exploratory nature of users tasks, the value of potential discoveries or the benefits of overall awareness. We need better metrics and benchmark repositories to compare tools, and we should also seek reports of successful adoption and demonstrated utility.

User Evaluation Methods for Visual Web Search Interfaces

“…Many researchers have proposed the use of inspection methods as part of their evaluation methodology for information visualization systems [23,25,34,35,38]. In general, there are two styles of inspection method: those that are based on a set of design guidelines, such as heuristic evaluations; and those that are based on performing specific tasks, such as cognitive walkthroughs.…”

“…The use of laboratory studies is promoted in most papers on the topic of evaluating information visualization systems [1,4,7,21,23,25,26,34]. In general, problems with such studies include the fact that they are time consuming, expensive, and difficult to design [7,34], and that participants are observed for a short period of time [23].…”

“…In general, problems with such studies include the fact that they are time consuming, expensive, and difficult to design [7,34], and that participants are observed for a short period of time [23]. Such studies often make use of students as participants, even though they may not accurately reflect the target user group [1,25].…”

“…This work builds upon the research by others on the topic of evaluating information visualization systems [1,4,7,16,23,25,26,29,34]. We present a cross-section of evaluation methods available for researchers to validate their designs for visual Web search interfaces.…”

“…However, it seems that few of these systems are being evaluated with human subjects in a systematic manner. The lack of empirical evidence regarding the utility of such interfaces makes it difficult for both academic and corporate researchers to evaluate the potential value that these interface improvements would provide to their systems and products [16,23]. As a result, few advancements have been adopted within the public Web search market.…”