Multimodal Gaze Interaction for Creative Design

Abstract

We present a new application ("Sakura") that enables people with physical impairments to produce creative visual design work using a multimodal gaze approach. The system integrates multiple features tailored for gaze interaction including the selection of design artefacts via a novel grid approach, control methods for manipulating canvas objects, creative typography, a new color selection approach, and a customizable guide technique facilitating the alignment of design elements. A user evaluation (N=24) found that nondisabled users were able to utilize the application to complete common design activities and that they rated the system positively in terms of usability. A follow-up study with physically impaired participants (N=6) demonstrated they were able to control the system when working towards a website design, rating the application as having a good level of usability. Our research highlights new directions in making creative activities more accessible for people with physical impairments.

Inclusive Voice Interaction Techniques for Creative Object Positioning

“…However, it will be crucial in future studies to investigate a range of different graphical objects (e.g. shapes, typography, smaller/larger objects, longer/shorter distances [9,18,20,36]) and how users find the experience of positioning objects in these different design scenarios. We feel given the positive findings from the research highlighted that the approaches would still provide an accessible experience in these different contexts, although this will be important to validate.…”

“…Adobe Photoshop [1] and Illustrator [2]). A similar approach has also been used in related work investigating the positioning of graphical objects within a design canvas via the combination of gaze interaction and mechanical switches to support people with physical impairment [9]. We adapted these approaches for use via voice control (Figure 3)users can initially enable the positional guide feature by stating "guides".…”

“…One crucial area where there has been a lack of work to date is around how graphical objects can efficiently be positioned around a digital canvas. A traditional mouse and keyboard input combination enables users to rapidly move digital objects around a design canvas such as nudging an object into a specific position or snapping to an alignment guide [9,21]. This is an essential component of controlling graphical design interfaces, yet it is unclear how this could be facilitated via voice interaction.…”

Voiceye

“…dwell time, speech commands, gaze gestures, etc. ), but we opted for mechanical switches that are commonly used by people with physical impairments [20]. There have been lack of studies investigating the use of switches as a selection approach in a multimodal gaze application, so this approach also presents an opportunity to explore users' perceptions of this type of interaction approach.…”

Voice Snapping: Inclusive Speech Interaction Techniques for Creative Object Manipulation

“…Snapping approaches (via mouse control) typically involve "smart" (sticky) snapping in alignment with other objects on the canvas or to guidelines that have been manually placed by users on the design canvas [7,9]. Research has shown the use of snapping can support alignment and unity within designs [10,24], thus supporting a designer's workfow and the production of professional outputs. However, similar to transformation handles, it is unclear whether object snapping can be benefcial where speech is the primary method of interaction to support people with physical impairments in manipulating creative objects.…”