Crowdlicit

Abstract

End-user elicitation studies are a popular design method. Currently, such studies are usually confined to a lab, limiting the number and diversity of participants, and therefore the representativeness of their results. Furthermore, the quality of the results from such studies generally lacks any formal means of evaluation. In this paper, we address some of the limitations of elicitation studies through the creation of the Crowdlicit system along with the introduction of end-user identification studies, which are the reverse of elicitation studies. Crowdlicit is a new web-based system that enables researchers to conduct online and in-lab elicitation and identification studies. We used Crowdlicit to run a crowd-powered elicitation study based on Morris's "Web on the Wall" study (2012) with 78 participants, arriving at a set of symbols that included six new symbols different from Morris's. We evaluated the effectiveness of 49 symbols (43 from Morris and six from Crowdlicit) by conducting a crowd-powered identification study. We show that the Crowdlicit elicitation study resulted in a set of symbols that was significantly more identifiable than Morris's. CCS CONCEPTS • Human-centered computing → HCI design and evaluation methods KEYWORDS End-user elicitation study; end-user identification study; user-driven design; crowdsourcing; Mechanical Turk.

Move Your Body: Engaging Museum Visitors with Human-Data Interaction

“…Ultimately, we believe that future work should investigate if and how elicitation studies should move out of research labs. In a different application context, Ali et al suggests running "crowd-powered elicitation study" [2]. In museum settings, researchers need to craft strategies for conducting elicitation studies in-situ: researchers may want to assess if conducting elicitation studies in-situ, with actual museum visitors (rather than in-lab, generally with participants recruited in an academic setting), will produce gestures and body movements that are more "intuitive" and easier to be discovered.…”

User-Defined Mid-Air Haptic Sensations for Interacting with an AR Menu Environment

“…Finally, one way to further validate the results of an elicitation study is what Ali et al [1] describe as end-user identification studies. They are the conceptual inverse of elicitation studies: they present participants with a symbol and ask which referent would be invoked by it.…”

Function Follows Form: Using the Aesthetic Association Principle to Enhance Haptic Interface Design

“…Villarreal-Narvaez et al (2020) provide an extensive review of gesture elicitation studies. A similar approach was used by Ali et al (2019) but added an additional evaluation loop and fed back user-elicited symbols in a separate study back to the participants with the task to identify the most fitting representatives for specific certain UI categories. Carbon (2019) emphasized that evaluation of novel and unfamiliar products requires user familiarization to gain valid data on longer-termed user experiences.…”