Criteria for Rigor in Visualization Design Study

Abstract

We develop a new perspective on research conducted through visualization design study that emphasizes design as a method of inquiry and the broad range of knowledge-contributions achieved through it as multiple, subjective, and socially constructed. From this interpretivist position we explore the nature of visualization design study and develop six criteria for rigor. We propose that rigor is established and judged according to the extent to which visualization design study research and its reporting are INFORMED, REFLEXIVE, ABUNDANT, PLAUSIBLE, RESONANT, and TRANSPARENT. This perspective and the criteria were constructed through a four-year engagement with the discourse around rigor and the nature of knowledge in social science, information systems, and design. We suggest methods from cognate disciplines that can support visualization researchers in meeting these criteria during the planning, execution, and reporting of design study. Through a series of deliberately provocative questions, we explore implications of this new perspective for design study research in visualization, concluding that as a discipline, visualization is not yet well positioned to embrace, nurture, and fully benefit from a rigorous, interpretivist approach to design study. The perspective and criteria we present are intended to stimulate dialogue and debate around the nature of visualization design study and the broader underpinnings of the discipline.

Design Exposition Discussion Documents for Rich Design Discourse in Applied Visualization

“…The project was developed with external collaborators with whom we had previously worked and an organisation and datasets with which we were already familiar. Making emphatic claims around how DExDs might transfer to other contexts [30] is therefore problematic. Additionally we have argued, with evidence, that DExDs are particularly relevant to applied visualization projects where domain experts are fully inculcated within the process of design -in this case, to develop design proposals that address challenging tasks by overriding established design patterns.…”

Reading Traces: Scalable Exploration in Elastic Visualizations of Cultural Heritage Data

“…We think of this design study as an interpretative approach to visualization research, during which we make sense of the qualitative evaluation results as one possible facet of results [MD20]. Aspects of observer‐dependence make it difficult to investigate effects on explorative characteristics or the usefulness of a visualization for the development of research questions.…”

PAVED: Pareto Front Visualization for Engineering Design

“…As Sedlmair et al point out, contributions that make design studies useful for other visualization researchers focus on various aspects of the problem domain, a validated visualization tool, or reflections on design guidelines [Sed16]. Meyer and Dykes particularly stress that the knowledge acquired through a design study is highly subjective and needs to be viewed in the context of its generation [MD19]. Inspired by their proposed subdivision of contributions into three topics, we provide our reflections on 1) the problem domain, 2) visualization idioms, and 3) methodological guidelines.…”