EvalBench: A Software Library for Visualization Evaluation

Abstract

It is generally acknowledged in visualization research that it is necessary to evaluate visualization artifacts in order to provide empirical evidence on their effectiveness and efficiency as well as their usability and utility. However, the difficulties of conducting such evaluations still remain an issue. Apart from the required know-how to appropriately design and conduct user studies, the necessary implementation effort for evaluation features in visualization software is a considerable obstacle. To mitigate this, we present EvalBench, an easy-to-use, flexible, and reusable software library for visualization evaluation written in Java. We describe its design choices and basic abstractions of our conceptual architecture and demonstrate its applicability by a number of case studies. EvalBench reduces implementation effort for evaluation features and makes conducting user studies easier. It can be used and integrated with third-party visualization prototypes that need to be evaluated via loose coupling. EvalBench supports both, quantitative and qualitative evaluation methods such as controlled experiments, interaction logging, laboratory questionnaires, heuristic evaluations, and insight diaries.

Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments

“…Such frameworks already exist to support the creation of computer based lab experiments (e.g., Touchstone [64], EvalBench [2]) and for very specific research domains (HVTE [9]). They have also started to emerge for web-based studies.…”

GraphUnit: Evaluating Interactive Graph Visualizations Using Crowdsourcing

“…Additional answer types: Finally, as described in section 3.1, GraphUnit currently supports a range of numeric and ordinal answer types, similarly to EvalBench [AHR13]. We would like to extend the types of answers that tasks can require, particularly with responses that involve interactions with the visualization.…”

A crowdsourcing system for integrated and reproducible evaluation in scientific visualization

“…Due to the prominent role of evaluation, the importance of software that has been developed with the purpose to support this task is widely acknowledged. EvalBench has been proposed as a flexible software library that aims at conducting user studies in an easier way [2]. EvalBench is written in Java and can be used to evaluate visualization applications in laboratory experiments.…”