Cuenesics

Abstract

Figure 1. People performing mid-air gestures to select items on interactive public displays. The system was installed at three locations: (L1) on a large wall-projection in a coworking space, (L2) on a rear-projection screen in a students' cafeteria, (L3) on an LCD at a venue's opening event ABSTRACTMost of today's public displays only show predefined contents that passers-by are not able to change. We argue that interactive public displays would benefit from immediately usable mid-air techniques for choosing options, expressing opinions or more generally selecting one among several items. We propose a design space for hand-gesture based mid-air selection techniques on interactive public displays, along with four specific techniques that we evaluated at three different locations in the the field. Our findings include: 1) if no hint is provided, people successfully use Point+Dwell for selecting items, 2) the user representation could be switched from Mirror to Cursor after registration without causing confusion, 3) people tend to explore items before confirming one, 4) in a public context, people frequently interact inadvertently (without looking at the screen). We conclude by providing recommendations for designers of interactive public displays to support immediate usability for mid-air selection.

Child-display interaction

“…Among them, touchless gestural interaction limits vandalism by placing displays in unreachable places [31], keeping a high hygiene level of the screen surface [19], and removing constraints to the display size (see, for instance, works on media façades [9]). Walter et al [34] focused their work on describing existing solutions found in literature, for user representation in touchless gestural applications. Recently, other works focused on the use of silhouettes or avatars [18,23], since they have proved to be very effective in solving some common pervasive display issues, namely interaction blindness (i.e.…”

“…Indeed, an appropriate interface could strongly contribute to address both the interaction blindness and affordance blindness and could make the interaction itself more intuitive and straightforward. Many prior works suggested the use of avatar-based interfaces, where a predominant human-shaped entity continuously reproduces user movements [18,23,33,34]. These studies revealed the effectiveness of silhouettes, mirror images or avatars in communicating the supported interactivity and its touchless nature.…”

Which one is me?

“…Prior work investigated different ways of representing users on public displays: Appearance matching representations include silhouettes [2] (aka contours [31,34]) and mirrors [26] ( Figures 1C and 1D). Kinestheticvisual matching ones include abstract objects [19,35,37], avatars [26], and skeletons [2,35,36] ( Figures 1A and 1B).…”

“…The past years witnessed an extensive employment of user representations on the display. These representations can serve multiple purposes; previous work employed them to attract the attention of passersby [26,36], initiate interaction with the display [35], and provide real-time feedback to users [1]. In most of these works, multi-user interaction is supported by showing multiple on-screen representations simultaneously.…”

Exploring the Use of Gesture in Collaborative Tasks

“…Integrating the use of both speech and gesture in computer understanding has been an ongoing challenge since the introduction of the point and manipulate "Putthat-there" system by Bolt [1]. A large amount of work has focused on designing and developing gesture-based interaction systems ranging from mobile systems [2,9,10] to large-screen displays [14,19,20], as well as researching the relationships used in multimodal and gestural interaction [6][7][8]12,[15][16][17][18].…”