An Ontological Framework for Supporting the Design and Evaluation of Visual Analytics Systems

Abstract

Designing, evaluating, and improving visual analytics (VA) systems is a primary area of activities in our discipline. In this paper, we present an ontological framework for recording and categorizing technical shortcomings to be addressed in a VA workflow, reasoning about the causes of such problems, identifying technical solutions, and anticipating secondary effects of the solutions. The methodology is built on the theoretical premise that designing a VA workflow is an optimization of the cost‐benefit ratio of the processes in the workflow. It makes uses three fundamental measures to group and connect “symptoms”, “causes”, “remedies”, and “side‐effects”, and guide the search for potential solutions to the problems. In terms of requirement analysis and system design, the proposed methodology can enable system designers to explore the decision space in a structured manner. In terms of evaluation, the proposed methodology is time‐efficient and complementary to various forms of empirical studies, such as user surveys, controlled experiments, observational studies, focus group discussions, and so on. In general, it reduces the amount of trial‐and‐error in the lifecycle of VA system development.

The Validity, Generalizability and Feasibility of Summative Evaluation Methods in Visual Analytics

“…Another set of rational studies, which are more sophisticated, rely on information theory premises [13]. Most remarkably, these works present an abstraction activity for solutions that are not abstract by nature [12,14]. The three activities we report for rational methods do not introduce any risk to the validity and generalizability criteria.…”

“…This framework treats the pipeline of generating and consuming visual artifacts as a communication channel that communicates information from raw data, as the sender, to human perception as the receiver. Information theory framework has been used to define objective metrics such as the cost-benefit ratio [15], which has been recently used to build an ontological framework that supports the design and evaluation of VA systems [12]. We include human-independent methods in our analysis because they are summative by nature.…”

A Bounded Measure for Estimating the Benefit of Visualization (Part I): Theoretical Discourse and Conceptual Evaluation

“…The above example demonstrates that using the KL-divergence to estimate PD can differentiate the knowledge variation between doctors and patients regarding the two pieces of knowledge that may reduce the distortion due to the "flat area". When it is used in Equation (1) in a relative or qualitative context (e.g., [36,37]), the unboundedness of the KL-divergence does not pose an issue. However, this does become an issue when the KL-divergence is used to measure PD in an absolute and quantitative context.…”

A Bounded Measure for Estimating the Benefit of Visualization (Part II): Case Studies and Empirical Evaluation

“…If we can address this mathematical issue successfully, we will be able to complement qualitative methods for assessing the value of visualization (e.g., by Wall et al [ 44 ]) with quantitative measurement; we will be able to carry out many experiments (e.g., those by Cleveland and McGill [ 35 ] and Saket et al [ 45 ]) to examine the trade-off between alphabet compression and potential distortion [ 3 ]; we will be able to estimate the knowledge used (or gained) by the users in (or from) visualization as discussed by Sacha et al [ 46 ]; and we will be able to transform the current qualitative methods for optimizing visual analytics workflow (e.g., [ 47 ]) to quantitative methods.…”