“… - used crowdsourcing only for the pilot study (e.g., [TGH12]);
- used crowdsourcing not for evaluation but for other purposes such as data collection (e.g., user generated layouts [KDMW16, KWD14, vHR08]);
- evaluated graphics but no information visualization (e.g., [BCER14,XADR13,KKL16]);
- evaluated user interfaces but no information visualization (e.g., [DCS*17,MBB*11,RYM*13]);
- evaluated human‐computer interaction but no information visualization (e.g., [MMS*08, KWS*14,DH08]);
- proposed a novel crowdsourcing platform (e.g., [TBRA17,EKR16]);
- used already available data maintained by a crowdsourcing platform without using crowdsourcing (e.g., [KHA16,DDW11]);
- discussed previous crowdsourcing studies without using crowdsourcing (e.g., [AJB16, ZGB*17,KH16]);
- mentioned the use of crowdsourcing for future evaluation (e.g., [HLS16]).
…”