Crowdsourcing for Information Visualization: Promises and Pitfalls

Self Cite

Visualizing network data is applicable in domains such as biology, engineering, and social sciences. We report the results of a study comparing the effectiveness of the two primary techniques for showing network data: node-link diagrams and adjacency matrices. Specifically, an evaluation with a large number of online participants revealed statistically significant differences between the two visualizations. Our work adds to existing research in several ways. First, we explore a broad spectrum of network tasks, many of which had not been previously evaluated. Second, our study uses two large datasets, typical of many real-life networks not explored by previous studies. Third, we leverage crowdsourcing to evaluate many tasks with many participants. This paper is an expanded journal version of a Graph Drawing (GD'17) conference paper. We evaluated a second dataset, added a qualitative feedback section, and expanded the procedure, results, discussion, and limitations sections.

Section: Methodsmentioning

confidence: 99%

Node-Link or Adjacency Matrices: Old Question, New Insights

Okoe

Jianu

Kobourov

2019

Self Cite

“…Our experimental comparison of visualization design alternatives continues a tradition established by Cleveland and McGill [22] and continued by the crowdsourced graphical perception work of Heer and Bostock [31]. Our experiment also makes use of a crowdsourcing platform, which serves to circumvent some of the limitations of directly observed lab studies while providing a diverse and large participant pool [12,13]. Our work follows two recent crowdsourced experiments relating to visualization on mobile phones: our previous study comparing alternative ways to visualize ranges on mobile phones [17], and Schwab et al's study of panning and zooming techniques with temporal data [54].…”

Section: (Mobile) Visualization Evaluation Studiesmentioning

confidence: 99%

A Comparative Evaluation of Animation and Small Multiples for Trend Visualization on Mobile Phones

Brehmer

Lee

Isenberg

et al. 2020

Self Cite

Fig. 1. Cropped screen captures from our experimental mobile application. Left: an instance of the animation condition (open this PDF in Acrobat Reader to view the animation). Right: an instance of the small multiples condition, in which the position of each item reflects its values in the year 2000; a "trail" encodes the item's previous positions from 1975 to 1999.Abstract-We compare the efficacy of animated and small multiples variants of scatterplots on mobile phones for comparing trends in multivariate datasets. Visualization is increasingly prevalent in mobile applications and mobile-first websites, yet there is little prior visualization research dedicated to small displays. In this paper, we build upon previous experimental research carried out on larger displays that assessed animated and non-animated variants of scatterplots. Incorporating similar experimental stimuli and tasks, we conducted an experiment where 96 crowdworker participants performed nine trend comparison tasks using their mobile phones. We found that those using a small multiples design consistently completed tasks in less time, albeit with slightly less confidence than those using an animated design. The accuracy results were more task-dependent, and we further interpret our results according to the characteristics of the individual tasks, with a specific focus on the trajectories of target and distractor data items in each task. We identify cases that appear to favor either animation or small multiples, providing new questions for further experimental research and implications for visualization design on mobile devices. Lastly, we provide a reflection on our evaluation methodology.

“…Following the crowdsourced graphical perception work of Heer and Bostock [32], our experiment involved the use of a crowdsourcing platform, which helps to overcome the limitations of controlled lab studies as it provides a diverse and large participant pool [12]. To our knowledge, our work is the first to conduct a visualization evaluation study on participants' mobile phones leveraging a crowd platform.…”

Section: Visualization Evaluation Studiesmentioning

confidence: 99%

Visualizing Ranges over Time on Mobile Phones: A Task-Based Crowdsourced Evaluation

Brehmer

Lee

Isenberg

et al. 2019

Self Cite

In the first crowdsourced visualization experiment conducted exclusively on mobile phones, we compare approaches to visualizing ranges over time on small displays. People routinely consume such data via a mobile phone, from temperatures in weather forecasting apps to sleep and blood pressure readings in personal health apps. However, we lack guidance on how to effectively visualize ranges on small displays in the context of different value retrieval and comparison tasks, or with respect to different data characteristics such as periodicity, seasonality, or the cardinality of ranges. Central to our experiment is a comparison between two ways to lay out ranges: a more conventional linear layout strikes a balance between quantitative and chronological scale resolution, while a less conventional radial layout emphasizes the cyclicality of time and may prioritize discrimination between values at its periphery. With results from 87 crowd workers, we found that while participants completed tasks more quickly with linear layouts than with radial ones, there were few differences in terms of error rate between layout conditions. We also found that participants performed similarly with both layouts in tasks that involved comparing superimposed observed and average ranges.