Advancing User-Centered Evaluation of Visual Analytic Environments through Contests

Costello, Loura; Grinstein, Georges; Plaisant, Catherine; Scholtz, Jean

doi:10.1057/ivs.2009.16

Cited by 14 publications

(12 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, two recent studies on co-located synchronous collaboration [39,50] used such a combination of techniques to analyze group work and analysis processes. Effectiveness of group work was assessed in the first study by using a scoring mechanism from the VAST contest [16,39]. Effectiveness of group work is often not well represented by time and accuracy which are common metrics for usability studies.…”

Section: Cda: Methods and Examplesmentioning

confidence: 99%

“…However, we also found experiments in which parts of the analysis process were controlled and studied in a laboratory setting. In this section, we focus on techniques that individual researchers can use, as opposed to community-wide evaluation efforts such as the Visualization Contest or the Visual Analytics Challenge [16]. The Visual Analytics Challenge provides a useful collection of data sets and analysis problems that can be used in wider VDAR evaluations, and a repository of examples that demonstrate how other tools have been used to analyse the data.…”

Section: Vdar: Evaluation Questionsmentioning

confidence: 99%

See 1 more Smart Citation

Empirical Studies in Information Visualization: Seven Scenarios

Lam

Bertini

Isenberg

et al. 2012

IEEE Trans. Visual. Comput. Graphics

539

442

View full text Add to dashboard Cite

We take a new, scenario-based look at evaluation in information visualization. Our seven scenarios, evaluating visual data analysis and reasoning, evaluating user performance, evaluating user experience, evaluating environments and work practices, evaluating communication through visualization, evaluating visualization algorithms, and evaluating collaborative data analysis were derived through an extensive literature review of over 800 visualization publications. These scenarios distinguish different study goals and types of research questions and are illustrated through example studies. Through this broad survey and the distillation of these scenarios, we make two contributions. One, we encapsulate the current practices in the information visualization research community and, two, we provide a different approach to reaching decisions about what might be the most effective evaluation of a given information visualization. Scenarios can be used to choose appropriate research questions and goals and the provided examples can be consulted for guidance on how to design one's own study.

show abstract

Section: Cda: Methods and Examplesmentioning

confidence: 99%

Section: Vdar: Evaluation Questionsmentioning

confidence: 99%

Empirical Studies in Information Visualization: Seven Scenarios

Lam

Bertini

Isenberg

et al. 2012

IEEE Trans. Visual. Comput. Graphics

539

442

View full text Add to dashboard Cite

show abstract

“…It has been used to successfully solve the text analytics problems in the past two VAST Challenges [16], most recently winning an award [10].…”

Section: Using Analystʼs Workpacementioning

confidence: 99%

Analyst's Workspace: An embodied sensemaking environment for large, high-resolution displays

Andrews

North

2012

2012 IEEE Conference on Visual Analytics Science and Technology (VAST)

View full text Add to dashboard Cite

Distributed cognition and embodiment provide compelling models for how humans think and interact with the environment. Our examination of the use of large, high-resolution displays from an embodied perspective has lead directly to the development of a new sensemaking environment called Analyst's Workspace (AW). AW leverages the embodied resources made more accessible through the physical nature of the display to create a spatial workspace. By combining spatial layout of documents and other artifacts with an entity-centric, explorative investigative approach, AW aims to allow the analyst to externalize elements of the sensemaking process as a part of the investigation, integrated into the visual representations of the data itself. In this paper, we describe the various capabilities of AW and discuss the key principles and concepts underlying its design, emphasizing unique design principles for designing visual analytic tools for large, high-resolution displays.

show abstract

“…In addition to looking at measures of accuracy, one of our goals is to look at the utility of the visualizations and the analytic process implicit in the visual analytics tool. 24 The VAST Challenge submissions include a writeup of the solution, a text explanation with screen shots showing the analytic process and the visualizations used, and a short video showing a portion of the analytic process. While this is certainly not a substitute for actually using the system, it is not feasible to get software (especially research software), and have reviewers install it on their own systems.…”

Section: An Analysis Of the 2009 Vast Challenge Reviewsmentioning

confidence: 99%

Developing guidelines for assessing visual analytics environments

Scholtz

2011

Information Visualization

Self Cite

View full text Add to dashboard Cite

In this article, we develop guidelines for evaluating visual analytics environments based on a synthesis of reviews for the entries to the 2009 Visual Analytics Science and Technology (VAST) Symposium Challenge and from a user study with professional intelligence analysts. By analyzing the 2009 VAST Challenge reviews, we gained a better understanding of what is important to our reviewers, both visualization researchers and professional analysts. We also report on a small user study with professional analysts to determine the important factors that they use in evaluating visual analysis systems. We also looked at guidelines developed by researchers in various domains and synthesized the results from these three efforts into an initial set for use by others in the community. One challenge for future visual analytics systems is to help in the generation of reports. In our user study, we also worked with analysts to understand the criteria they used to evaluate the quality of analytic reports. We propose that this knowledge will be useful as researchers look at systems to automate some of the report generation. 1 From these two efforts, we produced some initial guidelines for evaluating visual analytics environments and for the evaluation of analytic reports. It is important to understand that these guidelines are initial drafts and are limited in scope as the visual analytics systems we evaluated were used in specific tasks. We propose these guidelines as a starting point for the Visual Analytics Community.

show abstract

Advancing User-Centered Evaluation of Visual Analytic Environments through Contests

Cited by 14 publications

References 8 publications

Empirical Studies in Information Visualization: Seven Scenarios

Empirical Studies in Information Visualization: Seven Scenarios

Analyst's Workspace: An embodied sensemaking environment for large, high-resolution displays

Developing guidelines for assessing visual analytics environments

Contact Info

Product

Resources

About