Information Visualization Evaluation Using Crowdsourcing

Abstract

Visualization researchers have been increasingly leveraging crowdsourcing approaches to overcome a number of limitations of controlled laboratory experiments, including small participant sample sizes and narrow demographic backgrounds of study participants. However, as a community, we have little understanding on when, where, and how researchers use crowdsourcing approaches for visualization research. In this paper, we review the use of crowdsourcing for evaluation in visualization research. We analyzed 190 crowdsourcing experiments, reported in 82 papers that were published in major visualization conferences and journals between 2006 and 2017. We tagged each experiment along 36 dimensions that we identified for crowdsourcing experiments. We grouped our dimensions into six important aspects: study design & procedure, task type, participants, measures & metrics, quality assurance, and reproducibility. We report on the main findings of our review and discuss challenges and opportunities for improvements in conducting crowdsourcing studies for visualization research.

A Comparison of Radial and Linear Charts for Visualizing Daily Patterns

“…We found the highest number of incorrect responses for 12r (9%) and the lowest for 24l (2%). In four cases, users reported that the indicated values are equal when their value differences were rather low (1)(2)(3)(4)(5)(6)(7). This means, users underestimated the value differences.…”

“…Therefore, it is sensible to hypothesize that there is no benefit for a 24-hours radial chart, as it suffers from known decoding deficiencies, and at the same time does not provide the clock metaphor benefit of the 12-hours radial chart. As our goal was to investigate how well visual encodings are understood and accepted by a general audience, we crowd-sourced our experiment using Amazon's Mechanical Turk system, commonly used to perform graphical perception studies [7,31,38]. Also, previous studies comparing time series visualizations have been conducted remotely through Mechanical Turk [2,11].…”

“…As most of our tasks were rather simple to solve, we inspected the users' responses to decide whether to accept the HIT for the user. This is a common quality assurance procedure for crowd-sourced studies [7]. Upon completion of the study, the server displayed a code containing the session ID and an estimated error code, which users had to copy to the Mechanical Turk response field.…”

“…We set 100 users as target for our HIT, which is within the common range of participants in crowd-sourced studies [7] and was also used by the methodologically most similar prior study to ours by Brehmer et al [11]. We only allowed participants from North America.…”

A Comparative Evaluation of Animation and Small Multiples for Trend Visualization on Mobile Phones

“…Our experimental comparison of visualization design alternatives continues a tradition established by Cleveland and McGill [22] and continued by the crowdsourced graphical perception work of Heer and Bostock [31]. Our experiment also makes use of a crowdsourcing platform, which serves to circumvent some of the limitations of directly observed lab studies while providing a diverse and large participant pool [12,13]. Our work follows two recent crowdsourced experiments relating to visualization on mobile phones: our previous study comparing alternative ways to visualize ranges on mobile phones [17], and Schwab et al's study of panning and zooming techniques with temporal data [54].…”