Toward mixed method evaluations of scientific visualizations and design process as an evaluation tool

Abstract

In this position paper we discuss successes and limitations of current evaluation strategies for scientific visualizations and argue for embracing a mixed methods strategy of evaluation. The most novel contribution of the approach that we advocate is a new emphasis on employing design processes as practiced in related fields (e.g., graphic design, illustration, architecture) as a formalized mode of evaluation for data visualizations. To motivate this position we describe a series of recent evaluations of scientific visualization interfaces and computer graphics strategies conducted within our research group. Complementing these more traditional evaluations our visualization research group also regularly employs sketching, critique, and other design methods that have been formalized over years of practice in design fields. Our experience has convinced us that these activities are invaluable, often providing much more detailed evaluative feedback about our visualization systems than that obtained via more traditional user studies and the like. We believe that if design-based evaluation methodologies (e.g., ideation, sketching, critique) can be taught and embraced within the visualization community then these may become one of the most effective future strategies for both formative and summative evaluations.

A Systematic Review on the Practice of Evaluating Visualization

“…However, not only the judgment of domain experts but also that of visual experts such as artists, graphic designers, or illustrators can be useful as has been shown in a few cases [1,27,29,30]-in particular since 'critique' as a technique originated from teaching in the visual arts [35]. In visualization, it can be used in combination with techniques such as sketching and ideation when developing new visualization techniques [26]. For evaluating the impact of visualization tools on practices of real users, specific forms of case studies have been suggested [62].…”

Moving beyond sequential design: Reflections on a rich multi-channel approach to data visualization

“…There are some attempts to consider non-sequential refinements to the pipeline model, such as that of Jansen and Dragicevic [28], who account for visualization beyond the desktop to include branching and merging. Parallel development at this scale is more commonly discussed at the ideation phase of visualization design [27] where the cost of prototyping multiple designs is comparatively small. Some work reports on successful use of parallel prototyping [15] to inform visualization design [20], however multiple designs tend to be merged or filtered ultimately to inform a single 'product' or analytical objective.…”

Visualization evaluation for cyber security

“…• Critique: Holistic assessment of a human reading and interpretation of the visualization via "meticulous group discussion centered on how well particular aspects or details of a visualization support the intended goal" [19].…”