VA²: A Visual Analytics Approach for Evaluating Visual Analytics Applications

Abstract

Evaluation has become a fundamental part of visualization research and researchers have employed many approaches from the field of human-computer interaction like measures of task performance, thinking aloud protocols, and analysis of interaction logs. Recently, eye tracking has also become popular to analyze visual strategies of users in this context. This has added another modality and more data, which requires special visualization techniques to analyze this data. However, only few approaches exist that aim at an integrated analysis of multiple concurrent evaluation procedures. The variety, complexity, and sheer amount of such coupled multi-source data streams require a visual analytics approach. Our approach provides a highly interactive visualization environment to display and analyze thinking aloud, interaction, and eye movement data in close relation. Automatic pattern finding algorithms allow an efficient exploratory search and support the reasoning process to derive common eye-interaction-thinking patterns between participants. In addition, our tool equips researchers with mechanisms for searching and verifying expected usage patterns. We apply our approach to a user study involving a visual analytics application and we discuss insights gained from this joint analysis. We anticipate our approach to be applicable to other combinations of evaluation techniques and a broad class of visualization applications.

Characterizing Exploratory Visual Analysis: A Literature Review and Evaluation of Analytic Provenance in Tableau

“…Gotz and Wen [GW09] identify four common interaction sub‐sequences, which they use to generate visualization recommendations. Others compute n‐grams to identify common sub‐sequences [BJK∗16, BCS16]. Sequences are also used to build predictive (e.g., Markov) models, which can be used to compare analysts' performance [RJPL16], and predict users' future interactions [BCS16, DC17], task performance [GL12, BOZ∗14a], or personality traits [BOZ∗14a].…”

A Data Model and Task Space for Data of Interest (DOI) Eye-Tracking Analyses

“…Examples include scan paths, scarf plots, AOI rivers, and AOI transitions matrices. Research also exists on visual analytics principles and systems targeting AOI data, such as those by Andrienko et al [44], Weibel et al [45], Kurzhals et al [29], and Blascheck et al [30]. We will show that such methods do not scale to DOI data and are not suited for answering the types of questions that DOI data can answer.…”

“…This approach is particularly suited for evaluating the use of interactive visual content (e.g., interactive visualizations and visual analytic systems, computer games) a focus of increasing interest [30]. Should data enter or exit the screen, or be moved or reshaped through animation or interaction (e.g., a visual object is dragged to a different position), gazes continue to be correctly related to the updated state of the visualization.…”

“…Kurzhals et al and Blascheck et al recently pursued a visual analytics approach to the analysis of AOI data [29], [30]. Their systems support behavioral clustering and pattern searches, thus drawing closer to the systems that would be needed to support the questions described in Section 6.…”

A visual comparison of gaze behavior from pedestrians and cyclists

“…Multiple examples for using a visual analytics approach are available in the eye tracking literature [Andrienko et al 2012;Blascheck et al 2016c;Kurzhals et al 2014Kurzhals and Weiskopf 2013;Raschke et al 2014]. In their state-of-the-art report, Blascheck et al [2017] have collected many different visualization techniques for eye movement data.…”