A reflection on seven years of the VAST challenge

Scholtz, Jean; Whiting, Mark; Plaisant, Catherine; Grinstein, Georges

doi:10.1145/2442576.2442589

Cited by 17 publications

(15 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The VAST challenges provided realistic datasets and problem scenarios for researchers to evaluate their VA systems and problem-solving approaches [Scholtz et al 2012]. By solving these challenges, we were able to enrich the CZSaw script, explore the capabilities and limitations of the parametric mechanism, and suggest directions for further development.…”

Section: Use Casesmentioning

confidence: 99%

“…While investigating data, analysts generate unexpected questions that should either be investigated immediately or revisited later. To support iterative analysis by enabling users to review, retrieve, and revisit visualization states [Shneiderman 1996], history mechanisms such as undo or "time travel" enable revisitation in a variety of applications [Callahan et al 2006;Derthick and Roth 2001;Heer et al 2008;Kreuseler et al 2004;Shipman and Hsieh 2000]. Although history-related tools can play an important part in the visualization process, the best history mechanisms for achieving these benefits are not always clear.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Employing a Parametric Model for Analytic Provenance

Chen

Qian

Woodbury

et al. 2014

ACM Trans. Interact. Intell. Syst.

View full text Add to dashboard Cite

We introduce a propagation-based parametric symbolic model approach to supporting analytic provenance. This approach combines a script language to capture and encode the analytic process and a parametrically controlled symbolic model to represent and reuse the logic of the analysis process. Our approach first appeared in a visual analytics system called CZSaw. Using a script to capture the analyst's interactions at a meaningful system action level allows the creation of a parametrically controlled symbolic model in the form of a Directed Acyclic Graph (DAG). Using the DAG allows propagating changes. Graph nodes correspond to variables in CZSaw scripts, which are results (data and data visualizations) generated from user interactions. The user interacts with variables representing entities or relations to create the next step's results. Graph edges represent dependency relationships among nodes. Any change to a variable triggers the propagation mechanism to update downstream dependent variables and in turn updates data views to reflect the change. The analyst can reuse parts of the analysis process by assigning new values to a node in the graph. We evaluated this symbolic model approach by solving three IEEE VAST Challenge contest problems (from IEEE VAST 2008, 2009. In each of these challenges, the analyst first created a symbolic model to explore, understand, analyze, and solve a particular subproblem and then reused the model via its dependency graph propagation mechanism to solve similar subproblems. With the script and model, CZSaw supports the analytic provenance by capturing, encoding, and reusing the analysis process. The analyst can recall the chronological states of the analysis process with the CZSaw script and may interpret the underlying rationale of the analysis with the symbolic model.

show abstract

Section: Use Casesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Employing a Parametric Model for Analytic Provenance

Chen

Qian

Woodbury

et al. 2014

ACM Trans. Interact. Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…Plaisant et al [22] also recommend using benchmarks and contests to perform insight-based evaluations. These benchmarks have been used in contests that were popular in the InfoVis Community in 2003-2005, and then moved to the VAST Community with the VAST Challenge 2 organized every year since 2006 [5,23]. These contests remain a remarkable asset of the visualization community to share evaluation results and collect realistic benchmarks to test systems and engage researchers.…”

Section: Efforts In the Visualization Communitymentioning

confidence: 99%

Evaluating Visual Data Analysis Systems

Battle

Angelini

Binnig

et al. 2018

Proceedings of the Workshop on Human-in-the-Loop Data Analytics

View full text Add to dashboard Cite

Visual data analysis is a key tool for helping people to make sense of and interact with massive data sets. However, existing evaluation methods (e.g., database benchmarks, individual user studies) fail to capture the key points that make systems for visual data analysis (or visual data systems) challenging to design. In November 2017, members of both the Database and Visualization communities came together in a Dagstuhl seminar to discuss the grand challenges in the intersection of data analysis and interactive visualization. In this paper, we report on the discussions of the working group on the evaluation of visual data systems, which addressed questions centered around developing better evaluation methods, such as "How do the different communities evaluate visual data systems?" and "What we could learn from each other to develop evaluation techniques that cut across areas?". In their discussions, the group brainstormed initial steps towards new joint evaluation methods and developed a first concrete initiative-a trace repository of various real-world workloads and visual data systems-that enables researchers to derive evaluation setups (e.g., performance benchmarks, user studies) under more realistic assumptions, and enables new evaluation perspectives (e.g., broader meta analysis across analysis contexts, reproducibility and comparability across systems).

show abstract

“…Challenges such as YOHO [3], MNIST [4], HPC Challenge [5], ImageNet [6] and VAST [7], [8] have played important roles in driving progress in fields as diverse as machine learning, high performance computing and visual analytics. YOHO is the Linguistic Data Consortium database for voice verification systems and has been a critical enabler of speech research.…”

Section: Introductionmentioning

confidence: 99%

Static graph challenge: Subgraph isomorphism

Samsi

Gadepally

Hurley

et al. 2017

2017 IEEE High Performance Extreme Computing Conference (HPEC)

View full text Add to dashboard Cite

The rise of graph analytic systems has created a need for ways to measure and compare the capabilities of these systems. Graph analytics present unique scalability difficulties. The machine learning, high performance computing, and visual analytics communities have wrestled with these difficulties for decades and developed methodologies for creating challenges to move these communities forward. The proposed Subgraph Isomorphism Graph Challenge draws upon prior challenges from machine learning, high performance computing, and visual analytics to create a graph challenge that is reflective of many real-world graph analytics processing systems. The Subgraph Isomorphism Graph Challenge is a holistic specification with multiple integrated kernels that can be run together or independently. Each kernel is well defined mathematically and can be implemented in any programming environment. Subgraph isomorphism is amenable to both vertex-centric implementations and array-based implementations (e.g., using the Graph-BLAS.org standard). The computations are simple enough that performance predictions can be made based on simple computing hardware models. The surrounding kernels provide the context for each kernel that allows rigorous definition of both the input and the output for each kernel. Furthermore, since the proposed graph challenge is scalable in both problem size and hardware, it can be used to measure and quantitatively compare a wide range of present day and future systems. Serial implementations in C++, Python, Python with Pandas, Matlab, Octave, and Julia have been implemented and their single threaded performance have been measured. Specifications, data, and software are publicly available at GraphChallenge.org.

show abstract

A reflection on seven years of the VAST challenge

Cited by 17 publications

References 8 publications

Employing a Parametric Model for Analytic Provenance

Employing a Parametric Model for Analytic Provenance

Evaluating Visual Data Analysis Systems

Static graph challenge: Subgraph isomorphism

Contact Info

Product

Resources

About