Evaluating Visual Data Analysis Systems

Abstract

Visual data analysis is a key tool for helping people to make sense of and interact with massive data sets. However, existing evaluation methods (e.g., database benchmarks, individual user studies) fail to capture the key points that make systems for visual data analysis (or visual data systems) challenging to design. In November 2017, members of both the Database and Visualization communities came together in a Dagstuhl seminar to discuss the grand challenges in the intersection of data analysis and interactive visualization. In this paper, we report on the discussions of the working group on the evaluation of visual data systems, which addressed questions centered around developing better evaluation methods, such as "How do the different communities evaluate visual data systems?" and "What we could learn from each other to develop evaluation techniques that cut across areas?". In their discussions, the group brainstormed initial steps towards new joint evaluation methods and developed a first concrete initiative-a trace repository of various real-world workloads and visual data systems-that enables researchers to derive evaluation setups (e.g., performance benchmarks, user studies) under more realistic assumptions, and enables new evaluation perspectives (e.g., broader meta analysis across analysis contexts, reproducibility and comparability across systems).

Database Benchmarking for Supporting Real-Time Interactive Querying of Large Data

“…In this work, we present initial steps towards an end-to-end performance benchmark for interactive, real-time querying scenarios, derived from user study data that we collected for crossfilter contexts (a representative of dynamic queries). Our benchmark design is inspired by recent proposals [6,17] and tutorials [28,29,61] across multiple communities, and incorporates methodology from HCI, visualization, and databases. We ran our benchmark with 128 workflows and five different DBMSs, and found that none of these systems could adequately support our benchmark workload for datasets considered far from large in the database community.…”

“…This subsection defines key HCI and visualization terms related to the user study, summarized from prior work (e.g., [6,9,24,34]): A goal is a high level concept, representing an end state or result that the user wishes to achieve. To accomplish a goal, the user executes a set of tasks, which may include solving a specific problem or selecting among different alternatives.…”

“…Each user session was video recorded (both with screen capture and camera recording) and logged through the visual environment, implementing a multi-trace analysis model well described in previous work [6,9]. Participants' interactions are recorded in 3 main resources: application logs, containing information about the task itself (userID, taskID, dataset confidence, task confidence, answer, answer confidence, eventual comment); view logs, containing information about the configuration of the visualization environment for the given dataset; finally detailed log entries of participants' interactions with the visual environment during a task, collecting information like type of interaction (e.g., mousemove, brush filter, mouseout), timestamp, attribute on which the interaction happens, and mouse coordinates.…”

“…We envision a cross-community effort to develop benchmarks for a wide range of interactive use cases. Inspired by previous proposals (e.g., [6]), we are working towards a more general benchmark repository and platform, to execute data-driven benchmark development on a larger scale. Then anyone in the community could contribute new inputs (user logs, input datasets, interface designs), and the platform could derive benchmarks accordingly.…”

Performance model’s development: a novel approach encompassing ontology-based data access and visual analytics

“…We propose a novel approach to explore different/alternative definitions of the performance models while looking at their differences, hypothesize and test new performance models, and illustrate the results of these analyses in an interactive platform (see Battle et al 2018).…”